-- Logs begin at Tue 2022-09-27 03:40:37 UTC, end at Tue 2022-09-27 04:41:30 UTC. -- Sep 27 04:05:33.407459 dv-cinder-859186-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Sep 27 04:05:33.416116 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Sep 27 04:05:33.419765 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: *** Starting uWSGI 2.0.18-debian (64bit) on [Tue Sep 27 04:05:33 2022] *** Sep 27 04:05:33.419765 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Sep 27 04:05:33.419765 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Sep 27 04:05:33.419765 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: nodename: dv-cinder-859186-3-iscsi Sep 27 04:05:33.419765 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: machine: x86_64 Sep 27 04:05:33.419765 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: clock source: unix Sep 27 04:05:33.419765 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: pcre jit disabled Sep 27 04:05:33.419765 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: detected number of CPU cores: 4 Sep 27 04:05:33.419765 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: current working directory: / Sep 27 04:05:33.419765 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: detected binary path: /usr/bin/uwsgi-core Sep 27 04:05:33.419765 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: your processes number limit is 47680 Sep 27 04:05:33.419765 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: your memory page size is 4096 bytes Sep 27 04:05:33.420387 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: detected max file descriptor number: 2048 Sep 27 04:05:33.420387 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: lock engine: pthread robust mutexes Sep 27 04:05:33.420387 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: thunder lock: enabled Sep 27 04:05:33.420387 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Sep 27 04:05:33.420387 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Sep 27 04:05:33.459390 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: Python main interpreter initialized at 0x5603daf8b9c0 Sep 27 04:05:33.459390 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: python threads support enabled Sep 27 04:05:33.459390 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: your server socket listen backlog is limited to 100 connections Sep 27 04:05:33.459390 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: your mercy for graceful operations on workers is 90 seconds Sep 27 04:05:33.459749 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: mapped 403077 bytes (393 KB) for 2 cores Sep 27 04:05:33.459829 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: *** Operational MODE: preforking *** Sep 27 04:05:33.460082 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: *** uWSGI is running in multiple interpreter mode *** Sep 27 04:05:33.460082 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: spawned uWSGI master process (pid: 81412) Sep 27 04:05:33.460288 dv-cinder-859186-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Sep 27 04:05:33.460860 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: spawned uWSGI worker 1 (pid: 81420, cores: 1) Sep 27 04:05:33.461307 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: spawned uWSGI worker 2 (pid: 81421, cores: 1) Sep 27 04:05:33.461558 dv-cinder-859186-3-iscsi devstack@keystone.service[81412]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Sep 27 04:05:37.733370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 27 04:05:37.733370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: from cryptography.utils import int_from_bytes Sep 27 04:05:37.734189 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 27 04:05:37.734189 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: from cryptography.utils import int_from_bytes Sep 27 04:05:37.734307 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 27 04:05:37.734307 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: from cryptography.utils import int_from_bytes Sep 27 04:05:37.734713 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 27 04:05:37.734713 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: from cryptography.utils import int_from_bytes Sep 27 04:05:37.784156 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 27 04:05:37.784347 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 27 04:05:37.784455 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 27 04:05:37.784545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] command line args: [] {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 27 04:05:37.784622 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 27 04:05:37.784699 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] command line args: [] {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 27 04:05:37.784775 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 27 04:05:37.784959 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ================================================================================ {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 27 04:05:37.785043 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 27 04:05:37.785119 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ================================================================================ {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 27 04:05:37.785220 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] admin_token = **** {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.785370 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] admin_token = **** {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.785459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] config_dir = [] {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.785611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] config_dir = [] {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.785690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.785819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.785908 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] config_source = [] {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.786003 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] config_source = [] {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.786086 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] debug = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.786721 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] debug = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.786843 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.788034 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.788143 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.788215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.788281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.788345 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.788408 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.788471 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.788537 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.788596 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.788653 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] list_limit = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.788714 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] list_limit = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.788772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] log_config_append = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.788838 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] log_config_append = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.788902 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.788965 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.789023 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] log_dir = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.789079 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] log_dir = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.789135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] log_file = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.789195 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] log_file = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.789259 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.789323 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.789386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.789448 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.789506 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.789561 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.789624 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.789685 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.790153 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.790260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.790334 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.790405 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.790471 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.790535 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.790598 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.790661 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.790727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.790795 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.790853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.790914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.790976 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.791038 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.791229 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.791316 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.791447 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.791533 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.791689 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.791777 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.791956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.792049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.792435 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.26/identity {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.792520 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.26/identity {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.792604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] publish_errors = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.792676 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] publish_errors = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.792831 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.792914 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.793057 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.793144 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.793222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.793298 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.793461 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.793557 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.793693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.793810 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.793954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] standard_threads = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.794037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] standard_threads = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.794180 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.794268 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.794419 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.794502 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.794574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.794643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.794754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] use_journal = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.794884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] use_journal = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.794959 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] use_json = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.795035 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] use_json = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.795106 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] use_stderr = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.795176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] use_stderr = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.795246 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] use_syslog = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.795324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] use_syslog = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.795456 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.795535 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 04:05:37.795707 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.795790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.795941 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.796024 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.796192 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.796268 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.796425 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.796502 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.796669 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.796758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.796903 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.796981 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.797147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.797231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.797377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.797462 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.797620 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.797698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.797898 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.797980 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.798127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.798222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.798363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.798447 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.798603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.798679 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.798834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.798924 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.799075 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.799153 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.799317 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.799394 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.799550 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.799637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.799798 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.799875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.800048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.800124 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.800286 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.800439 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.800514 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.800585 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.800753 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.800831 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.801010 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.801097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.801257 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.801344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.801500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.801578 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.801728 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.802433 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.802512 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.802593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.802736 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.802814 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.802969 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.803047 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.803211 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.803287 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.803455 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.803544 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.803681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.803760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.803906 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.803990 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.804147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.804225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.804388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.804467 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.804623 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.804700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.804864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.804941 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.805093 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.805172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.805326 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.805403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.805553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.805630 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.805803 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.805886 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.806032 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.806110 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.806261 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.806348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.806500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.806574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.806704 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.806788 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.806948 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.807036 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.807173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.807252 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.807404 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.807481 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.807630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.807707 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.807862 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.807939 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.808089 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.808172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.808333 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.808411 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.808556 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.808664 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.808788 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.808893 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.809021 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.809124 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.809233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.809329 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.809467 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.809609 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.809753 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.809873 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.809955 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.810028 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.810214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.810316 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.810391 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.810520 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.810645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.810722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.810839 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.810928 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.811077 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.811164 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.811305 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.811393 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.811532 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.811617 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.811765 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.811844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.811994 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.812086 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.812257 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.812336 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.812491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.812619 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.812755 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.812899 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.813031 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.813154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.813295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.813435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.813559 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.813691 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.813864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.813997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.814127 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.814255 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.814379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.814514 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.814641 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.814765 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.814891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.815033 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.815109 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.815190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.815263 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.815335 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.815453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.815529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.815601 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.815682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.815756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.815828 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.815899 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.815970 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.816097 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.816186 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.816352 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.816432 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.816587 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.816665 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.816831 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.816924 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.817063 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.817143 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.817296 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.817374 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.817557 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.817710 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.817818 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.817893 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.818034 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.818113 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.818268 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.818355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.818493 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.818572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.818720 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.818836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.818947 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.819026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.819178 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.819258 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.819404 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.819483 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.819640 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.819717 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.819869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.819949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.820114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.820191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.820344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.820427 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.820595 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.820679 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.820845 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.820930 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.821081 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.821190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.821308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] auth.external = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.821389 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] auth.external = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.821535 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.821615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.821846 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.821932 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.822077 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.822170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.822307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] auth.password = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.822386 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] auth.password = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.822531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] auth.token = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.822610 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] auth.token = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.822768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.822846 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.822997 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.823077 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.823238 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.823319 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.823467 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.823546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.823709 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.823790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.823952 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.824030 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.824177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.824255 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.824408 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] credential.caching = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.824487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] credential.caching = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.824647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.824735 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.824890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.824971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.825135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.825215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.825368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.825453 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.825606 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.825685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.825860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.825941 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.826099 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.826179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.826347 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.826427 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.826581 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.826662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.826826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.826904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.827101 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.827180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.827340 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.827422 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.827596 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.827675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.827833 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.827911 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.828079 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.828159 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.828310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] federation.caching = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.828390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] federation.caching = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.828543 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.828621 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.828781 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.828861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.829020 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.829106 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.829246 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.829325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.829487 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.829567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.829724 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.829999 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.830078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.830235 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.830316 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.830476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.830554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.830711 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.830789 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.830949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.831034 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.831185 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] identity.caching = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.831264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.831414 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] identity.caching = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.831493 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.831658 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.831740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.831891 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.831970 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.832121 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.832199 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.832352 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.832433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.832583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.832660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.832810 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.832888 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.833061 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.833142 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.833309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.833388 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.833540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.833619 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.833791 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.833874 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.834022 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.834102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.834251 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.834404 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.834495 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.834570 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.834725 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.834804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.834967 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.835047 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.835210 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.835291 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.835450 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.835529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.835699 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.835778 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.835934 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.836014 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.836161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.836240 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.836385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.836463 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.836614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.836693 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.836838 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.836918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.837070 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.837150 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.837304 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.837384 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.837535 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.837615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.837785 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.837865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.838017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.838097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.838259 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.838338 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.838490 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.838568 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.838720 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.838800 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.838961 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.839047 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.839188 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.839274 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.839419 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.839498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.839643 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.839724 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.839865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.839952 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.840096 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.840176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.840327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.840407 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.840559 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.840646 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.840789 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.840904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.841127 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.841207 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.841380 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.841467 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.841619 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.841697 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.841871 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.841952 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.842102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.842194 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.842349 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.842429 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.842585 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.842663 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.842816 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.842902 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.843050 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.843136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.843280 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.843359 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.user = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.843510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.user = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.843598 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.843749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.843829 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.843984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.844062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.844218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.844304 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.844463 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.844542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.844705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.844783 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.844938 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.845025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.845168 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.845249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.845393 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.845471 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.845624 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.845710 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.845876 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.845957 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.846106 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.846184 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.846331 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.846421 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.846572 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.846658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.846814 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.846894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.847055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.847142 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.847293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.847371 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.847521 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.847598 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.847749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.847835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.847985 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.848063 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.848215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.848294 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.848447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.848533 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.849189 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.849274 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.849431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.849510 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.849671 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.849777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.849932 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.850011 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.850162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.850240 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.850403 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.850494 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.850628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.850707 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.850867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.850951 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.851095 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.851182 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.851326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.851405 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.851556 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.851635 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.851797 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.851884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.852025 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.852104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.852250 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.852329 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.852483 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.852571 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] resource.caching = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.852728 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.852970 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] resource.caching = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.853054 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.853203 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.853282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.853452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.853533 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.853682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.853778 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.853951 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.854031 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.854197 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.854276 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.854425 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.854504 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.854662 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.854813 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.854890 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] role.caching = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.854962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.855122 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.855201 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] role.driver = sql {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.855350 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.855436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] role.caching = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.855589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] role.driver = sql {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.855668 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.855815 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.855895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.856060 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.856148 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.856301 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.856381 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.856543 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.856622 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.856782 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.856869 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.857024 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.857102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.857262 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.857341 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.857492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.857578 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.857733 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.857858 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.857985 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.858064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.858227 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.858318 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.858471 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.858550 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.858712 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.858861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.858940 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.859099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.859333 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.859413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.859575 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.859655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.859821 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.859907 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.860068 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.860137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.860303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.860414 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.860553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.860636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.860787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.860868 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.861028 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.861106 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.861260 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.861340 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.861492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.861572 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.861718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.861834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.861972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.862054 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.862202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.862281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.862448 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.862527 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.862689 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.862767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.862926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.863005 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.863156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.863234 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.863389 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] token.caching = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.863468 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.863613 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.863691 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.863845 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] token.caching = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.863925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.864078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.864158 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.864364 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.864959 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.865112 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.865192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.865351 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.865439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.865592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.865672 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.865848 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.865928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.866089 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.866170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.866322 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.866400 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.866562 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.866640 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.866790 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.866870 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.867025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.867104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.867256 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.867350 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.867500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.867588 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.867742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.867829 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.867977 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.868064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.868216 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.868306 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] database.connection = **** {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.868476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.868568 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.868719 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.868797 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] database.connection = **** {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.868956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.869034 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.869193 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.869280 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.869424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.869503 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.869657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.869736 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.869910 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.870000 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.870143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.870222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.870374 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.870453 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.870605 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.870692 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.870844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.870923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.871076 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.871154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.871314 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.871400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.871544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.871623 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.871786 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.871871 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.872013 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.872099 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.872251 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.872329 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.872476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.872555 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.872627 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81420) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 27 04:05:37.872707 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.872931 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 04:05:37.873099 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81421) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 27 04:05:37.901685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:37.901938 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:37.902117 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:37.902201 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:37.902350 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:37.902573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:37.910429 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:37.910774 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:37.939089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:37.939317 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:37.939729 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:37.939959 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:37.992412 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:37.992575 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.040308 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.040538 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.041058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.041289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.194189 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.194410 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.194524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.194635 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.194756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.194863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.194977 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.195062 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.195195 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.195277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.195409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.195620 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.218512 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.218795 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.218952 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.219036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.219188 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.219273 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.219417 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.219500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.219632 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.219723 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.219848 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.219931 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.220060 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.220273 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.231447 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.231832 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.244330 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.244517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.244605 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.244776 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.244860 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.245000 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.245083 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.245220 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.245303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.245437 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.245521 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.245653 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.245736 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81420) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.245890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81421) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 04:05:38.262881 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.262982 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.263093 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 27 04:05:38.263178 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 27 04:05:38.263439 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.263523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.263645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 27 04:05:38.263729 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 27 04:05:38.263925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.264008 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.264190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.264281 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.264447 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.264530 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.264706 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.264786 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.274783 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.274891 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.275060 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.275144 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.275327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.275410 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.275606 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.275691 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.282585 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81421) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.282683 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81420) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.285467 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81421) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.285572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81420) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.285776 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.285863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.286057 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.286140 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.286888 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.286995 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.287173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.287257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.287435 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.287518 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.287705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.287790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.287979 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.288059 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.288245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.288334 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.288517 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.288596 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.288782 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.288862 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.311513 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.311640 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.312964 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81421) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.313109 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81420) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.313247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.313394 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.317985 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81421) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.318138 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81420) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.318267 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.318425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.318544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.318709 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.324889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81421) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.325086 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81420) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.325170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.325375 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.330742 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81421) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.330998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81420) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.331083 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.331296 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.331380 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.331584 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.331684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.331864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.331951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.332140 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.332222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.332408 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.332491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.332673 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.332755 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.332942 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.349486 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.349722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.349826 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.350190 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.353635 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81420) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.354587 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81421) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.356842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.357141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.357813 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.358110 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.361327 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81420) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.362464 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81421) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.364460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81420) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.365711 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81421) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.367643 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.367947 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.368237 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.368523 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.368804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.368941 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.369086 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.369240 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.369524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.369833 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.370119 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.370406 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.385188 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81420) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.385478 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.385761 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.386039 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.386594 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81421) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.386885 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.387155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.387420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.392897 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.394260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.394466 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.396310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.396553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.396834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.397865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.398142 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.399650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81420) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.399939 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.400226 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.400512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.400829 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81421) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.400912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.401114 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.401396 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.401680 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.402094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.412714 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81420) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.413002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.413283 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.413563 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.413974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.414071 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81421) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.414258 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.414341 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.414539 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.414621 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.414817 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.415193 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.415473 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.415767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.430074 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81420) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.431205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81421) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.432970 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81420) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.434108 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81421) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.435898 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.437270 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.437363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.438826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.438910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81420) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.439111 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.439518 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.440305 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81421) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.440587 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.440985 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.445975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.447427 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.447509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81420) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.448875 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81421) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.450352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.450753 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.451042 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.451322 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.451806 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.452202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.452488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.452764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.459707 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81420) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.460000 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.460280 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.461114 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81421) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.461404 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.461684 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.467214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81420) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.467500 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.467782 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.468061 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.468339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.468560 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81421) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 04:05:38.468643 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.468844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.468926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.469139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.469220 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.469405 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.469496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.469679 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.469788 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.469978 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.470068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.470241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.470321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.470508 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.470589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.470782 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.470906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81420) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.471067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.471346 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.471606 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.471870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.472148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81421) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 04:05:38.505684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x5603daf8b9c0 pid: 81420 (default app) Sep 27 04:05:38.506799 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x5603daf8b9c0 pid: 81421 (default app) Sep 27 04:05:38.508910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d40c4d-f254-4677-b23b-4b85db891fa6 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:38.509194 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d40c4d-f254-4677-b23b-4b85db891fa6 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:38.509466 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d40c4d-f254-4677-b23b-4b85db891fa6 None None] PATH_INFO: `/v3` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:38.513588 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1/1] 10.222.0.26 () {50 vars in 791 bytes} [Tue Sep 27 04:05:38 2022] GET /identity/v3/ => generated 255 bytes in 7 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:05:45.245177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c35f095-ca01-43df-ac68-6d40f7a67ccc None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:45.245693 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c35f095-ca01-43df-ac68-6d40f7a67ccc None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:45.245693 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c35f095-ca01-43df-ac68-6d40f7a67ccc None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:45.249963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1/2] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:05:45 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:05:45.255486 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a37ac9a-0493-421d-aa37-54fcb908406a None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:45.256018 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a37ac9a-0493-421d-aa37-54fcb908406a None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:45.256018 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a37ac9a-0493-421d-aa37-54fcb908406a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:45.325397 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: INFO dbcounter [None req-2a37ac9a-0493-421d-aa37-54fcb908406a None None] Registered counter for database keystone Sep 27 04:05:45.344354 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG oslo_db.sqlalchemy.engines [None req-2a37ac9a-0493-421d-aa37-54fcb908406a None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81420) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 27 04:05:45.351151 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writer thread running {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 27 04:05:45.535099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG passlib.handlers.bcrypt [None req-2a37ac9a-0493-421d-aa37-54fcb908406a None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81420) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 27 04:05:45.535401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG passlib.handlers.bcrypt [None req-2a37ac9a-0493-421d-aa37-54fcb908406a None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81420) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 27 04:05:45.592831 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-2a37ac9a-0493-421d-aa37-54fcb908406a None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:05:45.594431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-2a37ac9a-0493-421d-aa37-54fcb908406a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:05:45.720041 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 2/3] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:05:45 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 465 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 27 04:05:46.175975 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c35f095-ca01-43df-ac68-6d40f7a67ccc None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:46.245178 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: INFO dbcounter [None req-1c35f095-ca01-43df-ac68-6d40f7a67ccc None None] Registered counter for database keystone Sep 27 04:05:46.266865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG oslo_db.sqlalchemy.engines [None req-1c35f095-ca01-43df-ac68-6d40f7a67ccc None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81421) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 27 04:05:46.273022 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writer thread running {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 27 04:05:46.389902 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c35f095-ca01-43df-ac68-6d40f7a67ccc None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:46.394173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee017266-d5dc-4116-b94e-27726dc2c978 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:46.395320 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee017266-d5dc-4116-b94e-27726dc2c978 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:46.395320 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee017266-d5dc-4116-b94e-27726dc2c978 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:46.395519 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee017266-d5dc-4116-b94e-27726dc2c978 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:46.405493 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee017266-d5dc-4116-b94e-27726dc2c978 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:05:46.457968 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: WARNING keystone.common.rbac_enforcer.enforcer [None req-ee017266-d5dc-4116-b94e-27726dc2c978 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 27 04:05:46.458285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee017266-d5dc-4116-b94e-27726dc2c978 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:05:46.468115 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 2/4] 10.222.0.26 () {58 vars in 1324 bytes} [Tue Sep 27 04:05:46 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 04:05:46.476108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a37ac9a-0493-421d-aa37-54fcb908406a None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:46.493246 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a37ac9a-0493-421d-aa37-54fcb908406a None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:46.497626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1dc4eeb-3998-42c8-8aca-0dcdb2776f15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:46.498764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1dc4eeb-3998-42c8-8aca-0dcdb2776f15 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:46.499048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1dc4eeb-3998-42c8-8aca-0dcdb2776f15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:46.499332 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1dc4eeb-3998-42c8-8aca-0dcdb2776f15 admin admin] PATH_INFO: `/v3/projects/3cc5c8e99cb64893a8d035d6aa4f3eb6` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:46.501152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1dc4eeb-3998-42c8-8aca-0dcdb2776f15 admin admin] RBAC: Authorizing `identity:get_project(project_id=3cc5c8e99cb64893a8d035d6aa4f3eb6)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:05:46.554386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: WARNING keystone.common.rbac_enforcer.enforcer [None req-f1dc4eeb-3998-42c8-8aca-0dcdb2776f15 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 27 04:05:46.554736 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1dc4eeb-3998-42c8-8aca-0dcdb2776f15 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:05:46.559167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 3/5] 10.222.0.26 () {56 vars in 1207 bytes} [Tue Sep 27 04:05:46 2022] GET /identity/v3/projects/3cc5c8e99cb64893a8d035d6aa4f3eb6 => generated 344 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:05:48.678527 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d080d690-d57d-4c73-9d61-800bd6e744d2 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:48.679001 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d080d690-d57d-4c73-9d61-800bd6e744d2 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:48.679001 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d080d690-d57d-4c73-9d61-800bd6e744d2 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:48.683055 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 3/6] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:05:48 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:05:48.688371 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102ec156-39eb-41ce-8278-fcd54a1adfc1 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:48.688773 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102ec156-39eb-41ce-8278-fcd54a1adfc1 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:48.688928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102ec156-39eb-41ce-8278-fcd54a1adfc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:48.709648 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-102ec156-39eb-41ce-8278-fcd54a1adfc1 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:05:48.711151 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-102ec156-39eb-41ce-8278-fcd54a1adfc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:05:48.718762 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 4/7] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:05:48 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 27 04:05:48.992270 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d080d690-d57d-4c73-9d61-800bd6e744d2 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:49.004529 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d080d690-d57d-4c73-9d61-800bd6e744d2 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:49.009224 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2386cd0-418a-41f4-ae66-6b77c8b879bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:49.009854 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2386cd0-418a-41f4-ae66-6b77c8b879bd admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:49.010136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2386cd0-418a-41f4-ae66-6b77c8b879bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:49.010410 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2386cd0-418a-41f4-ae66-6b77c8b879bd admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:49.018177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2386cd0-418a-41f4-ae66-6b77c8b879bd admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:05:49.020152 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2386cd0-418a-41f4-ae66-6b77c8b879bd admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:05:49.029609 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 4/8] 10.222.0.26 () {58 vars in 1324 bytes} [Tue Sep 27 04:05:48 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 04:05:49.034910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102ec156-39eb-41ce-8278-fcd54a1adfc1 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:49.044513 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102ec156-39eb-41ce-8278-fcd54a1adfc1 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:49.049072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076ffbf7-de7d-4855-9888-6725d965eba9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:49.049692 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076ffbf7-de7d-4855-9888-6725d965eba9 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:49.050000 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076ffbf7-de7d-4855-9888-6725d965eba9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:49.050276 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076ffbf7-de7d-4855-9888-6725d965eba9 admin admin] PATH_INFO: `/v3/users/6190ca8b8acd4e99b228f0851e2ed5af` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:49.051568 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076ffbf7-de7d-4855-9888-6725d965eba9 admin admin] RBAC: Authorizing `identity:get_user(user_id=6190ca8b8acd4e99b228f0851e2ed5af)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:05:49.053913 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076ffbf7-de7d-4855-9888-6725d965eba9 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:05:49.055421 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 5/9] 10.222.0.26 () {56 vars in 1198 bytes} [Tue Sep 27 04:05:49 2022] GET /identity/v3/users/6190ca8b8acd4e99b228f0851e2ed5af => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:05:51.249213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7ff6bb-7f92-4ac9-bcc5-ba83e71e1a14 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:51.249820 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7ff6bb-7f92-4ac9-bcc5-ba83e71e1a14 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:51.249820 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7ff6bb-7f92-4ac9-bcc5-ba83e71e1a14 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:51.251554 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 5/10] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:05:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:05:51.256999 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5769d125-ba50-4b44-b3a4-b1f089ffda6f None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:51.257432 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5769d125-ba50-4b44-b3a4-b1f089ffda6f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:51.257528 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5769d125-ba50-4b44-b3a4-b1f089ffda6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:51.262532 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8909d028-9871-494a-a265-0c02690c0010 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:51.262822 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8909d028-9871-494a-a265-0c02690c0010 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:51.263207 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8909d028-9871-494a-a265-0c02690c0010 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:51.264417 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 6/11] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:05:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:05:51.269503 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2deb4cb-000d-4552-a5fa-08d3b1df003a None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:51.269825 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2deb4cb-000d-4552-a5fa-08d3b1df003a None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:51.270100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2deb4cb-000d-4552-a5fa-08d3b1df003a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:51.276775 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-5769d125-ba50-4b44-b3a4-b1f089ffda6f None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:05:51.277989 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-5769d125-ba50-4b44-b3a4-b1f089ffda6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:05:51.311922 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG passlib.handlers.bcrypt [None req-c2deb4cb-000d-4552-a5fa-08d3b1df003a None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81421) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 27 04:05:51.312302 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG passlib.handlers.bcrypt [None req-c2deb4cb-000d-4552-a5fa-08d3b1df003a None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81421) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 27 04:05:51.357049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 6/12] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:05:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:05:51.361676 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-c2deb4cb-000d-4552-a5fa-08d3b1df003a None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:05:51.363275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c2deb4cb-000d-4552-a5fa-08d3b1df003a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:05:51.461729 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 7/13] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:05:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:05:51.620197 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5769d125-ba50-4b44-b3a4-b1f089ffda6f None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:51.644160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:05:51.722798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5769d125-ba50-4b44-b3a4-b1f089ffda6f None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:51.725120 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2deb4cb-000d-4552-a5fa-08d3b1df003a None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:51.756750 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c8ac4e-9a4b-4dcb-b71c-b819552b2a97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:51.757379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c8ac4e-9a4b-4dcb-b71c-b819552b2a97 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:51.757658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c8ac4e-9a4b-4dcb-b71c-b819552b2a97 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:51.757965 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c8ac4e-9a4b-4dcb-b71c-b819552b2a97 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:51.769342 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c8ac4e-9a4b-4dcb-b71c-b819552b2a97 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:05:51.778070 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-39c8ac4e-9a4b-4dcb-b71c-b819552b2a97 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:05:51.778070 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:05:51.778070 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:05:51.778070 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:05:51.778070 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:05:51.778070 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:05:51.778070 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:05:51.778070 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:05:51.778070 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:05:51.778070 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:05:51.778070 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:05:51.778070 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:05:51.778070 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:05:51.778070 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:05:51.778070 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:05:51.778070 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:05:51.778974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:05:51.778974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:05:51.778974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:05:51.778974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:05:51.778974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:05:51.778974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:05:51.778974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:05:51.778974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:05:51.778974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:05:51.778974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:05:51.778974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:05:51.778974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:05:51.778974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:05:51.778974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:05:51.778974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:05:51.778974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:05:51.779793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:05:51.779793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:05:51.779793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:05:51.779793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:05:51.779793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:05:51.779793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:05:51.779793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:05:51.779793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:05:51.779793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:05:51.779793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:05:51.779793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:05:51.779793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:05:51.779793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 7/14] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:05:51 2022] GET /identity/v3/roles/admin => generated 83 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:05:51.784775 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c8ac4e-9a4b-4dcb-b71c-b819552b2a97 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:51.826491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2deb4cb-000d-4552-a5fa-08d3b1df003a None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:51.828775 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:05:51.864229 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:05:51.865275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075e99f9-2305-43a3-85b4-dc45e908b853 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:51.866174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075e99f9-2305-43a3-85b4-dc45e908b853 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:51.866484 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075e99f9-2305-43a3-85b4-dc45e908b853 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:51.866758 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075e99f9-2305-43a3-85b4-dc45e908b853 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:51.885271 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c8ac4e-9a4b-4dcb-b71c-b819552b2a97 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:51.918257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca998bf0-b9c0-46ca-8497-afe52f7de1e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:51.918858 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca998bf0-b9c0-46ca-8497-afe52f7de1e5 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:51.919140 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca998bf0-b9c0-46ca-8497-afe52f7de1e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:51.919413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca998bf0-b9c0-46ca-8497-afe52f7de1e5 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:51.920076 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca998bf0-b9c0-46ca-8497-afe52f7de1e5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:05:51.920380 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca998bf0-b9c0-46ca-8497-afe52f7de1e5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:05:51.921790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca998bf0-b9c0-46ca-8497-afe52f7de1e5 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:05:51.933270 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075e99f9-2305-43a3-85b4-dc45e908b853 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:05:51.933641 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 8/15] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:05:51 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:05:51.934882 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075e99f9-2305-43a3-85b4-dc45e908b853 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:05:51.939262 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca998bf0-b9c0-46ca-8497-afe52f7de1e5 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:52.029421 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 8/16] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:05:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 27 04:05:52.034150 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca998bf0-b9c0-46ca-8497-afe52f7de1e5 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:52.034883 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075e99f9-2305-43a3-85b4-dc45e908b853 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:52.038083 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:05:52.067524 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6798d1-12ec-41e0-a5b4-ea493827c7db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:52.068148 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6798d1-12ec-41e0-a5b4-ea493827c7db None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:52.068424 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6798d1-12ec-41e0-a5b4-ea493827c7db None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:52.068693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6798d1-12ec-41e0-a5b4-ea493827c7db None admin] PATH_INFO: `/v3/users/6190ca8b8acd4e99b228f0851e2ed5af` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:52.069840 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6798d1-12ec-41e0-a5b4-ea493827c7db None admin] RBAC: Authorizing `identity:get_user(user_id=6190ca8b8acd4e99b228f0851e2ed5af)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:05:52.071280 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6798d1-12ec-41e0-a5b4-ea493827c7db None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:05:52.072858 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 9/17] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:05:51 2022] GET /identity/v3/users/6190ca8b8acd4e99b228f0851e2ed5af => generated 247 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:05:52.078129 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6798d1-12ec-41e0-a5b4-ea493827c7db None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:52.092262 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:05:52.133584 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075e99f9-2305-43a3-85b4-dc45e908b853 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:52.170165 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36719ecc-c873-49f5-94fd-70f641f59b11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:52.170936 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36719ecc-c873-49f5-94fd-70f641f59b11 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:52.171212 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36719ecc-c873-49f5-94fd-70f641f59b11 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:52.171478 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36719ecc-c873-49f5-94fd-70f641f59b11 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:52.175415 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6798d1-12ec-41e0-a5b4-ea493827c7db None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:52.190051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36719ecc-c873-49f5-94fd-70f641f59b11 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:05:52.191581 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36719ecc-c873-49f5-94fd-70f641f59b11 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:05:52.209885 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08c5c33-1ab6-4542-bc76-e4a8a429eccc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:52.210487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-36719ecc-c873-49f5-94fd-70f641f59b11 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:05:52.210487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:05:52.210487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:05:52.210487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:05:52.210487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:05:52.210487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:05:52.210487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:05:52.210487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:05:52.210487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:05:52.210487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:05:52.210487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:05:52.210487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:05:52.210487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:05:52.210487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:05:52.210487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:05:52.210487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:05:52.211403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:05:52.211403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:05:52.211403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:05:52.211403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:05:52.211403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:05:52.211403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:05:52.211403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:05:52.211403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:05:52.211403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:05:52.211403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:05:52.211403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:05:52.211403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:05:52.211403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:05:52.211403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:05:52.211403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:05:52.211403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:05:52.211403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:05:52.212335 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:05:52.212335 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:05:52.212335 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:05:52.212490 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08c5c33-1ab6-4542-bc76-e4a8a429eccc None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:52.212490 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08c5c33-1ab6-4542-bc76-e4a8a429eccc None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:52.212490 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08c5c33-1ab6-4542-bc76-e4a8a429eccc None admin] PATH_INFO: `/v3/domains/default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:52.213123 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 9/18] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:05:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:05:52.213223 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b08c5c33-1ab6-4542-bc76-e4a8a429eccc None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:05:52.213572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b08c5c33-1ab6-4542-bc76-e4a8a429eccc None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:05:52.215133 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 10/19] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:05:52 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:05:52.217362 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36719ecc-c873-49f5-94fd-70f641f59b11 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:52.220661 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08c5c33-1ab6-4542-bc76-e4a8a429eccc None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:52.273503 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:05:52.299060 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:05:52.317883 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08c5c33-1ab6-4542-bc76-e4a8a429eccc None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:52.320599 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36719ecc-c873-49f5-94fd-70f641f59b11 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:52.351390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eee964a-77e9-49e9-9114-f2425cc053a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:52.351991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eee964a-77e9-49e9-9114-f2425cc053a4 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:52.352269 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eee964a-77e9-49e9-9114-f2425cc053a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:52.352540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eee964a-77e9-49e9-9114-f2425cc053a4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:52.353264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eee964a-77e9-49e9-9114-f2425cc053a4 None admin] RBAC: Adding query filter params (role.id=e6a31562a18649f0809202c36a8e18ef, scope.domain.id=default, user.id=6190ca8b8acd4e99b228f0851e2ed5af) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:05:52.353573 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eee964a-77e9-49e9-9114-f2425cc053a4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:05:52.354505 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2e95f2-1218-47ad-a9c0-08e5be548da3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:52.355042 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eee964a-77e9-49e9-9114-f2425cc053a4 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:05:52.355145 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2e95f2-1218-47ad-a9c0-08e5be548da3 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:52.355367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2e95f2-1218-47ad-a9c0-08e5be548da3 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:52.355637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2e95f2-1218-47ad-a9c0-08e5be548da3 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:52.356266 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2e95f2-1218-47ad-a9c0-08e5be548da3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:05:52.356564 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2e95f2-1218-47ad-a9c0-08e5be548da3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:05:52.357960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2e95f2-1218-47ad-a9c0-08e5be548da3 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:05:52.361620 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 11/20] 10.222.0.26 () {56 vars in 1322 bytes} [Tue Sep 27 04:05:52 2022] GET /identity/v3/role_assignments?user.id=6190ca8b8acd4e99b228f0851e2ed5af&scope.domain.id=default&role.id=e6a31562a18649f0809202c36a8e18ef => generated 233 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:05:52.376933 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 10/21] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:05:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:05:54.671873 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cb7aa7-96fe-4f77-9a7e-2f988d7cb306 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:54.672361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cb7aa7-96fe-4f77-9a7e-2f988d7cb306 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:54.672361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cb7aa7-96fe-4f77-9a7e-2f988d7cb306 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:54.673668 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 12/22] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:05:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:05:54.691821 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8044e019-7de3-408b-a0e9-593c5ee93853 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:54.693321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8044e019-7de3-408b-a0e9-593c5ee93853 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:54.694006 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8044e019-7de3-408b-a0e9-593c5ee93853 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:54.718754 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-8044e019-7de3-408b-a0e9-593c5ee93853 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:05:54.720312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-8044e019-7de3-408b-a0e9-593c5ee93853 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:05:54.799373 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 11/23] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:05:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:05:55.005154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827e5383-1407-4442-bf51-7a411453a3c1 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:55.005880 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827e5383-1407-4442-bf51-7a411453a3c1 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:55.006379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827e5383-1407-4442-bf51-7a411453a3c1 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:55.007924 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 13/24] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:05:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:05:55.013365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8b7aa5-908c-43a7-8b13-53abdef4c6ab None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:55.013907 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8b7aa5-908c-43a7-8b13-53abdef4c6ab None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:55.014401 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8b7aa5-908c-43a7-8b13-53abdef4c6ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:55.048067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-6f8b7aa5-908c-43a7-8b13-53abdef4c6ab None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:05:55.049569 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-6f8b7aa5-908c-43a7-8b13-53abdef4c6ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:05:55.109251 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:05:55.144330 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 12/25] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:05:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:05:55.338612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dcd59f-d7d8-462e-b28f-d0d0dce1a668 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:55.338790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dcd59f-d7d8-462e-b28f-d0d0dce1a668 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:55.339063 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dcd59f-d7d8-462e-b28f-d0d0dce1a668 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:55.340529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 14/26] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:05:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:05:55.349975 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c0a625-4f2c-4b73-b929-f9aec21431d3 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:55.350266 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c0a625-4f2c-4b73-b929-f9aec21431d3 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:55.350543 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c0a625-4f2c-4b73-b929-f9aec21431d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:55.362847 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dcd59f-d7d8-462e-b28f-d0d0dce1a668 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:55.378266 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-f9c0a625-4f2c-4b73-b929-f9aec21431d3 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:05:55.379475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-f9c0a625-4f2c-4b73-b929-f9aec21431d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:05:55.480295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 13/27] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:05:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:05:55.482830 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d20b3e4-9df7-42a5-8e49-010168784b81 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:55.483793 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d20b3e4-9df7-42a5-8e49-010168784b81 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:55.484379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d20b3e4-9df7-42a5-8e49-010168784b81 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:55.487937 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 14/28] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:05:55 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:05:55.493376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce32ef8-acae-4d1e-b442-e316247a8299 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:55.493901 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce32ef8-acae-4d1e-b442-e316247a8299 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:55.494324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce32ef8-acae-4d1e-b442-e316247a8299 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:55.524935 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-4ce32ef8-acae-4d1e-b442-e316247a8299 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:05:55.526557 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-4ce32ef8-acae-4d1e-b442-e316247a8299 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:05:55.557003 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dcd59f-d7d8-462e-b28f-d0d0dce1a668 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:55.626378 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7812284-3cd0-4102-b202-eee41ffb7c56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:55.626934 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7812284-3cd0-4102-b202-eee41ffb7c56 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:55.627215 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7812284-3cd0-4102-b202-eee41ffb7c56 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:55.627490 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7812284-3cd0-4102-b202-eee41ffb7c56 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:55.650991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7812284-3cd0-4102-b202-eee41ffb7c56 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:05:55.659416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-d7812284-3cd0-4102-b202-eee41ffb7c56 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:05:55.659416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:05:55.659416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:05:55.659416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:05:55.659416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:05:55.659416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:05:55.659416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:05:55.659416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:05:55.659416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:05:55.659416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:05:55.659416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:05:55.659416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:05:55.659416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:05:55.659416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:05:55.659416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:05:55.659416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:05:55.660435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:05:55.660435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:05:55.660435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:05:55.660435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:05:55.660435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:05:55.660435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:05:55.660435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:05:55.660435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:05:55.660435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:05:55.660435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:05:55.660435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:05:55.660435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:05:55.660435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:05:55.660435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:05:55.660435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:05:55.660435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:05:55.661975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:05:55.661975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:05:55.661975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:05:55.661975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:05:55.661975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:05:55.661975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:05:55.661975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:05:55.661975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:05:55.661975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:05:55.661975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:05:55.661975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:05:55.661975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:05:55.661975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 15/29] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:05:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 298 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:05:55.661975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7812284-3cd0-4102-b202-eee41ffb7c56 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:55.663723 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 15/30] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:05:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:05:55.666737 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:05:55.669304 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce32ef8-acae-4d1e-b442-e316247a8299 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:55.733036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:05:55.831159 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7812284-3cd0-4102-b202-eee41ffb7c56 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:55.868525 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99976c29-5af8-47ac-b24f-c8f83bbe27e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:55.869170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99976c29-5af8-47ac-b24f-c8f83bbe27e9 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:55.869453 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99976c29-5af8-47ac-b24f-c8f83bbe27e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:55.869729 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99976c29-5af8-47ac-b24f-c8f83bbe27e9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:55.876109 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce32ef8-acae-4d1e-b442-e316247a8299 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:55.881669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99976c29-5af8-47ac-b24f-c8f83bbe27e9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:05:55.884011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99976c29-5af8-47ac-b24f-c8f83bbe27e9 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:05:55.900866 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-99976c29-5af8-47ac-b24f-c8f83bbe27e9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:05:55.900866 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:05:55.900866 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:05:55.900866 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:05:55.900866 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:05:55.900866 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:05:55.900866 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:05:55.900866 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:05:55.900866 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:05:55.900866 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:05:55.900866 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:05:55.900866 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:05:55.900866 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:05:55.900866 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:05:55.900866 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:05:55.900866 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:05:55.901727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:05:55.901727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:05:55.901727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:05:55.901727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:05:55.901727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:05:55.901727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:05:55.901727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:05:55.901727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:05:55.901727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:05:55.901727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:05:55.901727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:05:55.901727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:05:55.901727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:05:55.901727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:05:55.901727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:05:55.901727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:05:55.901727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:05:55.902751 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:05:55.902751 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:05:55.902751 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:05:55.902751 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 16/31] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:05:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:05:55.903015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99976c29-5af8-47ac-b24f-c8f83bbe27e9 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:55.966337 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:05:55.980626 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd74239-2cb7-4049-bb85-5142e194a4c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:55.982309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd74239-2cb7-4049-bb85-5142e194a4c8 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:55.982641 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd74239-2cb7-4049-bb85-5142e194a4c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:55.982918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd74239-2cb7-4049-bb85-5142e194a4c8 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:55.983597 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd74239-2cb7-4049-bb85-5142e194a4c8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:05:55.983897 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd74239-2cb7-4049-bb85-5142e194a4c8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:05:55.985272 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd74239-2cb7-4049-bb85-5142e194a4c8 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:05:56.009910 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 16/32] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:05:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 341 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:05:56.010751 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd74239-2cb7-4049-bb85-5142e194a4c8 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:56.040432 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99976c29-5af8-47ac-b24f-c8f83bbe27e9 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:56.084625 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313193dd-40e1-4085-921d-053e69a3e5c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:56.085533 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313193dd-40e1-4085-921d-053e69a3e5c1 None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:56.085836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313193dd-40e1-4085-921d-053e69a3e5c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:56.086357 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313193dd-40e1-4085-921d-053e69a3e5c1 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:56.087455 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313193dd-40e1-4085-921d-053e69a3e5c1 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:05:56.089358 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313193dd-40e1-4085-921d-053e69a3e5c1 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:05:56.135264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:05:56.139469 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd74239-2cb7-4049-bb85-5142e194a4c8 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:56.180089 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd74c9d-c4a9-458c-8add-ba9ba03f4410 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:56.180696 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd74c9d-c4a9-458c-8add-ba9ba03f4410 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:56.180976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd74c9d-c4a9-458c-8add-ba9ba03f4410 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:56.181250 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd74c9d-c4a9-458c-8add-ba9ba03f4410 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:56.181936 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd74c9d-c4a9-458c-8add-ba9ba03f4410 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:05:56.182240 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd74c9d-c4a9-458c-8add-ba9ba03f4410 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:05:56.183608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd74c9d-c4a9-458c-8add-ba9ba03f4410 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:05:56.196252 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 17/33] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:05:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:05:56.197178 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd74c9d-c4a9-458c-8add-ba9ba03f4410 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:56.233045 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 17/34] 10.222.0.26 () {60 vars in 1131 bytes} [Tue Sep 27 04:05:55 2022] POST /identity/v3/roles => generated 225 bytes in 330 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:05:56.233949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313193dd-40e1-4085-921d-053e69a3e5c1 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:56.312068 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd74c9d-c4a9-458c-8add-ba9ba03f4410 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:56.352432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64747030-0a8d-433c-be00-7b6252914e88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:56.353085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64747030-0a8d-433c-be00-7b6252914e88 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:56.353368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64747030-0a8d-433c-be00-7b6252914e88 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:56.353644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64747030-0a8d-433c-be00-7b6252914e88 None admin] PATH_INFO: `/v3/users/6190ca8b8acd4e99b228f0851e2ed5af` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:56.355271 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64747030-0a8d-433c-be00-7b6252914e88 None admin] RBAC: Authorizing `identity:get_user(user_id=6190ca8b8acd4e99b228f0851e2ed5af)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:05:56.356894 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64747030-0a8d-433c-be00-7b6252914e88 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:05:56.358806 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 18/35] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:05:56 2022] GET /identity/v3/users/6190ca8b8acd4e99b228f0851e2ed5af => generated 247 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:05:56.359629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64747030-0a8d-433c-be00-7b6252914e88 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:56.392318 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:05:56.450956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313193dd-40e1-4085-921d-053e69a3e5c1 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:56.490302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:05:56.493612 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64747030-0a8d-433c-be00-7b6252914e88 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:56.502802 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47fc4f3-c9c8-40a8-8b28-b4a430df1d4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:56.503431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47fc4f3-c9c8-40a8-8b28-b4a430df1d4b None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:56.503718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47fc4f3-c9c8-40a8-8b28-b4a430df1d4b None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:56.503982 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47fc4f3-c9c8-40a8-8b28-b4a430df1d4b None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:56.504654 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a47fc4f3-c9c8-40a8-8b28-b4a430df1d4b None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:05:56.506608 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a47fc4f3-c9c8-40a8-8b28-b4a430df1d4b None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:05:56.528544 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 18/36] 10.222.0.26 () {60 vars in 1131 bytes} [Tue Sep 27 04:05:56 2022] POST /identity/v3/roles => generated 219 bytes in 295 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:05:56.530018 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47fc4f3-c9c8-40a8-8b28-b4a430df1d4b None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:56.539716 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e9d473-d71c-4bd5-bfad-92ce3aa6b1be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:56.541065 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e9d473-d71c-4bd5-bfad-92ce3aa6b1be None admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:56.541673 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e9d473-d71c-4bd5-bfad-92ce3aa6b1be None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:56.542307 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e9d473-d71c-4bd5-bfad-92ce3aa6b1be None admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:56.543306 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e9d473-d71c-4bd5-bfad-92ce3aa6b1be None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:05:56.545366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e9d473-d71c-4bd5-bfad-92ce3aa6b1be None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:05:56.703188 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 19/37] 10.222.0.26 () {60 vars in 1141 bytes} [Tue Sep 27 04:05:56 2022] POST /identity/v3/projects => generated 301 bytes in 344 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:05:56.717227 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47fc4f3-c9c8-40a8-8b28-b4a430df1d4b None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:56.766322 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41377ed-23db-4963-a1e0-22af54893ba6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:56.767329 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41377ed-23db-4963-a1e0-22af54893ba6 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:56.767826 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41377ed-23db-4963-a1e0-22af54893ba6 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:56.768325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41377ed-23db-4963-a1e0-22af54893ba6 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:56.769952 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e41377ed-23db-4963-a1e0-22af54893ba6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:05:56.772241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e41377ed-23db-4963-a1e0-22af54893ba6 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:05:56.774739 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 19/38] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:05:56 2022] GET /identity/v3/domains/default => generated 199 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:05:56.776071 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeaeab2-3082-4500-8f6c-8e026eba2781 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:56.776519 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeaeab2-3082-4500-8f6c-8e026eba2781 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:56.776707 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeaeab2-3082-4500-8f6c-8e026eba2781 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:56.782177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 20/39] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:05:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:05:56.784625 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41377ed-23db-4963-a1e0-22af54893ba6 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:56.789068 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e436f2-fd8c-40f3-8474-b5fde83f0863 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:56.789352 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e436f2-fd8c-40f3-8474-b5fde83f0863 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:56.789638 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e436f2-fd8c-40f3-8474-b5fde83f0863 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:56.822934 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-77e436f2-fd8c-40f3-8474-b5fde83f0863 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:05:56.824837 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-77e436f2-fd8c-40f3-8474-b5fde83f0863 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:05:56.844755 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:05:56.913088 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41377ed-23db-4963-a1e0-22af54893ba6 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:56.917655 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 21/40] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:05:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:05:56.953358 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbf374b-5b9d-4d33-829e-f635f38f01a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:56.954042 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbf374b-5b9d-4d33-829e-f635f38f01a1 None admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:56.954325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbf374b-5b9d-4d33-829e-f635f38f01a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:56.954604 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbf374b-5b9d-4d33-829e-f635f38f01a1 None admin] PATH_INFO: `/v3/domains/default/users/6190ca8b8acd4e99b228f0851e2ed5af/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:56.956557 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbf374b-5b9d-4d33-829e-f635f38f01a1 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=6190ca8b8acd4e99b228f0851e2ed5af, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:05:56.958809 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbf374b-5b9d-4d33-829e-f635f38f01a1 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:05:56.975632 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 20/41] 10.222.0.26 () {58 vars in 1361 bytes} [Tue Sep 27 04:05:56 2022] PUT /identity/v3/domains/default/users/6190ca8b8acd4e99b228f0851e2ed5af/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:05:56.999033 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2defff-64f8-4423-9448-bfcfded659ba None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:56.999655 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2defff-64f8-4423-9448-bfcfded659ba None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:57.000159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2defff-64f8-4423-9448-bfcfded659ba None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:57.002103 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 22/42] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:05:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:05:57.008953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a554279-042e-44d6-b211-6fc1d9db193e None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:57.009600 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a554279-042e-44d6-b211-6fc1d9db193e None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:57.010105 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a554279-042e-44d6-b211-6fc1d9db193e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:57.030233 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-1a554279-042e-44d6-b211-6fc1d9db193e None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:05:57.031804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-1a554279-042e-44d6-b211-6fc1d9db193e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:05:57.106375 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 21/43] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:05:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:05:57.190084 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2defff-64f8-4423-9448-bfcfded659ba None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:57.201637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:05:57.295460 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2defff-64f8-4423-9448-bfcfded659ba None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:57.329285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fa4395-9045-4425-865e-1dc7c2cfc162 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:57.329915 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fa4395-9045-4425-865e-1dc7c2cfc162 None admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:57.330195 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fa4395-9045-4425-865e-1dc7c2cfc162 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:57.330464 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fa4395-9045-4425-865e-1dc7c2cfc162 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:57.331124 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63fa4395-9045-4425-865e-1dc7c2cfc162 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:05:57.332616 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63fa4395-9045-4425-865e-1dc7c2cfc162 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:05:57.349286 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 23/44] 10.222.0.26 () {60 vars in 1131 bytes} [Tue Sep 27 04:05:57 2022] POST /identity/v3/roles => generated 223 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:05:57.381771 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a554279-042e-44d6-b211-6fc1d9db193e None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:57.417440 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:05:57.495608 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a554279-042e-44d6-b211-6fc1d9db193e None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:57.534305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc54b39b-fc51-47fa-a781-e096d265ccad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:57.534958 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc54b39b-fc51-47fa-a781-e096d265ccad None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:57.535527 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc54b39b-fc51-47fa-a781-e096d265ccad None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:57.535527 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc54b39b-fc51-47fa-a781-e096d265ccad None admin] PATH_INFO: `/v3/domains/default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:57.537046 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc54b39b-fc51-47fa-a781-e096d265ccad None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:05:57.538703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc54b39b-fc51-47fa-a781-e096d265ccad None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:05:57.539700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 22/45] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:05:57 2022] GET /identity/v3/domains/default => generated 199 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:05:57.545333 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fa4395-9045-4425-865e-1dc7c2cfc162 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:57.619482 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:05:57.654158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fa4395-9045-4425-865e-1dc7c2cfc162 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:57.686666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27569ffd-a74a-473f-b6ad-49d621a0906f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:57.687285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27569ffd-a74a-473f-b6ad-49d621a0906f None admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:57.687568 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27569ffd-a74a-473f-b6ad-49d621a0906f None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:57.687846 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27569ffd-a74a-473f-b6ad-49d621a0906f None admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:57.688524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27569ffd-a74a-473f-b6ad-49d621a0906f None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:05:57.690014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27569ffd-a74a-473f-b6ad-49d621a0906f None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:05:57.717190 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 24/46] 10.222.0.26 () {60 vars in 1141 bytes} [Tue Sep 27 04:05:57 2022] POST /identity/v3/projects => generated 312 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:05:59.160443 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9664de-464d-4a98-8252-c6909c24854c None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:59.160959 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9664de-464d-4a98-8252-c6909c24854c None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:59.161069 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9664de-464d-4a98-8252-c6909c24854c None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:59.162344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 23/47] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:05:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:05:59.167699 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40887f99-025b-4f8e-ba5f-085f56ff7fe8 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:59.168080 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40887f99-025b-4f8e-ba5f-085f56ff7fe8 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:59.168262 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40887f99-025b-4f8e-ba5f-085f56ff7fe8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:59.186335 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-40887f99-025b-4f8e-ba5f-085f56ff7fe8 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:05:59.187505 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-40887f99-025b-4f8e-ba5f-085f56ff7fe8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:05:59.256926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 25/48] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:05:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:05:59.518897 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9664de-464d-4a98-8252-c6909c24854c None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:59.723597 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9664de-464d-4a98-8252-c6909c24854c None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:59.726089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:05:59.761367 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7095ac-4287-462f-841f-b44d9841d24d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:59.762027 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7095ac-4287-462f-841f-b44d9841d24d None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:59.762308 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7095ac-4287-462f-841f-b44d9841d24d None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:59.762585 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7095ac-4287-462f-841f-b44d9841d24d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:59.773080 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7095ac-4287-462f-841f-b44d9841d24d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:05:59.777910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-ca7095ac-4287-462f-841f-b44d9841d24d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:05:59.777910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:05:59.777910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:05:59.777910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:05:59.777910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:05:59.777910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:05:59.777910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:05:59.777910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:05:59.777910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:05:59.777910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:05:59.777910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:05:59.777910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:05:59.777910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:05:59.777910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:05:59.777910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:05:59.777910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:05:59.778734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:05:59.778734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:05:59.778734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:05:59.778734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:05:59.778734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:05:59.778734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:05:59.778734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:05:59.778734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:05:59.778734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:05:59.778734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:05:59.778734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:05:59.778734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:05:59.778734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:05:59.778734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:05:59.778734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:05:59.778734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:05:59.779477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:05:59.779477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:05:59.779477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:05:59.779477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:05:59.779477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:05:59.779477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:05:59.779477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:05:59.779477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:05:59.779477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:05:59.779477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:05:59.779477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:05:59.779477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:05:59.779477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 24/49] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:05:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:05:59.784790 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40887f99-025b-4f8e-ba5f-085f56ff7fe8 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:05:59.832564 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:05:59.839095 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353bedd9-3d1d-40bf-8218-3df80bddca38 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:59.839383 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353bedd9-3d1d-40bf-8218-3df80bddca38 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:59.839663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353bedd9-3d1d-40bf-8218-3df80bddca38 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:59.841017 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 25/50] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:05:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:05:59.845968 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b319f6e5-7209-4928-aedf-4d9a078c1a65 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:59.846296 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b319f6e5-7209-4928-aedf-4d9a078c1a65 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:59.846574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b319f6e5-7209-4928-aedf-4d9a078c1a65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:59.864855 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-b319f6e5-7209-4928-aedf-4d9a078c1a65 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:05:59.866272 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-b319f6e5-7209-4928-aedf-4d9a078c1a65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:05:59.883418 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40887f99-025b-4f8e-ba5f-085f56ff7fe8 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:05:59.917928 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ccfd30-f0f0-42c8-bce6-c9698fb53fd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:05:59.918540 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ccfd30-f0f0-42c8-bce6-c9698fb53fd4 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:05:59.918825 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ccfd30-f0f0-42c8-bce6-c9698fb53fd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:05:59.919101 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ccfd30-f0f0-42c8-bce6-c9698fb53fd4 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:05:59.919775 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ccfd30-f0f0-42c8-bce6-c9698fb53fd4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:05:59.920083 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ccfd30-f0f0-42c8-bce6-c9698fb53fd4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:05:59.921462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ccfd30-f0f0-42c8-bce6-c9698fb53fd4 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:05:59.930084 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 26/51] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:05:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:05:59.931949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 26/52] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:05:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:05:59.936480 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ccfd30-f0f0-42c8-bce6-c9698fb53fd4 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:00.035343 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ccfd30-f0f0-42c8-bce6-c9698fb53fd4 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:00.048222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:00.068838 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19de022-9751-49bf-920b-fc3636340832 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:00.069457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19de022-9751-49bf-920b-fc3636340832 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:00.069750 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19de022-9751-49bf-920b-fc3636340832 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:00.070038 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19de022-9751-49bf-920b-fc3636340832 None admin] PATH_INFO: `/v3/users/6190ca8b8acd4e99b228f0851e2ed5af` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:00.071148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19de022-9751-49bf-920b-fc3636340832 None admin] RBAC: Authorizing `identity:get_user(user_id=6190ca8b8acd4e99b228f0851e2ed5af)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:00.072574 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19de022-9751-49bf-920b-fc3636340832 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:00.074348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 27/53] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:05:59 2022] GET /identity/v3/users/6190ca8b8acd4e99b228f0851e2ed5af => generated 247 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:00.079560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b319f6e5-7209-4928-aedf-4d9a078c1a65 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:00.168274 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:00.177490 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b319f6e5-7209-4928-aedf-4d9a078c1a65 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:00.201056 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19de022-9751-49bf-920b-fc3636340832 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:00.211133 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca1fd53-04c4-40c1-928c-2fcfc21c2538 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:00.211787 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca1fd53-04c4-40c1-928c-2fcfc21c2538 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:00.212071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca1fd53-04c4-40c1-928c-2fcfc21c2538 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:00.212349 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca1fd53-04c4-40c1-928c-2fcfc21c2538 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:00.213498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca1fd53-04c4-40c1-928c-2fcfc21c2538 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:00.215025 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca1fd53-04c4-40c1-928c-2fcfc21c2538 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:00.216653 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 27/54] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:06:00 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:00.222753 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca1fd53-04c4-40c1-928c-2fcfc21c2538 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:00.329947 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca1fd53-04c4-40c1-928c-2fcfc21c2538 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:00.364068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25ed68b-b37d-4140-b68c-6b0d0bb04de2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:00.364694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25ed68b-b37d-4140-b68c-6b0d0bb04de2 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:00.364983 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25ed68b-b37d-4140-b68c-6b0d0bb04de2 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:00.365257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25ed68b-b37d-4140-b68c-6b0d0bb04de2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:00.366008 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25ed68b-b37d-4140-b68c-6b0d0bb04de2 None admin] RBAC: Adding query filter params (role.id=e6a31562a18649f0809202c36a8e18ef, scope.domain.id=default, user.id=6190ca8b8acd4e99b228f0851e2ed5af) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:00.366314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25ed68b-b37d-4140-b68c-6b0d0bb04de2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:00.367767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25ed68b-b37d-4140-b68c-6b0d0bb04de2 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:00.373310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 28/55] 10.222.0.26 () {56 vars in 1322 bytes} [Tue Sep 27 04:06:00 2022] GET /identity/v3/role_assignments?user.id=6190ca8b8acd4e99b228f0851e2ed5af&scope.domain.id=default&role.id=e6a31562a18649f0809202c36a8e18ef => generated 530 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:00.414979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19de022-9751-49bf-920b-fc3636340832 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:00.450311 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74530371-587a-4d2b-abb2-e12c3be97419 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:00.450960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74530371-587a-4d2b-abb2-e12c3be97419 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:00.451242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74530371-587a-4d2b-abb2-e12c3be97419 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:00.451516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74530371-587a-4d2b-abb2-e12c3be97419 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:00.452671 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74530371-587a-4d2b-abb2-e12c3be97419 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:00.454169 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74530371-587a-4d2b-abb2-e12c3be97419 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:00.456293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 28/56] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:06:00 2022] GET /identity/v3/domains/default => generated 199 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:00.462650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25ed68b-b37d-4140-b68c-6b0d0bb04de2 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:00.489205 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:00.572085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25ed68b-b37d-4140-b68c-6b0d0bb04de2 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:00.607538 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022c52e5-9e35-4d91-bb6a-8c305db4a231 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:00.608077 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022c52e5-9e35-4d91-bb6a-8c305db4a231 None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:00.608356 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022c52e5-9e35-4d91-bb6a-8c305db4a231 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:00.608636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022c52e5-9e35-4d91-bb6a-8c305db4a231 None admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:00.609297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022c52e5-9e35-4d91-bb6a-8c305db4a231 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:00.610805 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022c52e5-9e35-4d91-bb6a-8c305db4a231 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:00.639000 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 29/57] 10.222.0.26 () {60 vars in 1140 bytes} [Tue Sep 27 04:06:00 2022] POST /identity/v3/projects => generated 298 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:06:02.761427 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c97e4c-5a14-4b3b-86b0-2d736c3d7e0e None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:02.761949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c97e4c-5a14-4b3b-86b0-2d736c3d7e0e None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:02.762033 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c97e4c-5a14-4b3b-86b0-2d736c3d7e0e None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:02.763341 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 29/58] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:02.768363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e11abc-e64a-4c8f-91f7-442035034dbc None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:02.768791 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e11abc-e64a-4c8f-91f7-442035034dbc None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:02.768919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e11abc-e64a-4c8f-91f7-442035034dbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:02.786417 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-26e11abc-e64a-4c8f-91f7-442035034dbc None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:02.787591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-26e11abc-e64a-4c8f-91f7-442035034dbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:02.850220 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 30/59] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:03.116840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c97e4c-5a14-4b3b-86b0-2d736c3d7e0e None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:03.165274 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:03.214237 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c97e4c-5a14-4b3b-86b0-2d736c3d7e0e None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:03.249350 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52dbf4ff-a035-431d-b6d4-450601cd8a3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:03.249999 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52dbf4ff-a035-431d-b6d4-450601cd8a3b None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:03.250283 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52dbf4ff-a035-431d-b6d4-450601cd8a3b None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:03.250558 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52dbf4ff-a035-431d-b6d4-450601cd8a3b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:03.251908 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52dbf4ff-a035-431d-b6d4-450601cd8a3b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:03.253338 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52dbf4ff-a035-431d-b6d4-450601cd8a3b None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:03.254927 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 30/60] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:06:03 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:03.260355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e11abc-e64a-4c8f-91f7-442035034dbc None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:03.268160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:03.355309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e11abc-e64a-4c8f-91f7-442035034dbc None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:03.386899 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64fb131-8335-4b79-87c0-7bf5cd68cc4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:03.387434 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64fb131-8335-4b79-87c0-7bf5cd68cc4c None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:03.387710 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64fb131-8335-4b79-87c0-7bf5cd68cc4c None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:03.387983 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64fb131-8335-4b79-87c0-7bf5cd68cc4c None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:03.388618 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64fb131-8335-4b79-87c0-7bf5cd68cc4c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:03.390069 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64fb131-8335-4b79-87c0-7bf5cd68cc4c None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:03.415971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 31/61] 10.222.0.26 () {60 vars in 1132 bytes} [Tue Sep 27 04:06:03 2022] POST /identity/v3/users => generated 275 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:06:06.720149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae3f9c9-7bd8-4ac3-8541-d5b6f83ec8e1 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:06.721125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae3f9c9-7bd8-4ac3-8541-d5b6f83ec8e1 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:06.721541 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae3f9c9-7bd8-4ac3-8541-d5b6f83ec8e1 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:06.723046 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 31/62] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:06.735479 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd53c47d-91b0-4c71-a106-8e50418b8f3c None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:06.736287 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd53c47d-91b0-4c71-a106-8e50418b8f3c None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:06.736753 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd53c47d-91b0-4c71-a106-8e50418b8f3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:06.761158 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-fd53c47d-91b0-4c71-a106-8e50418b8f3c None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:06.762826 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-fd53c47d-91b0-4c71-a106-8e50418b8f3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:06.808753 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394bf1b9-d191-48df-ae60-5d3e4eca1bcf None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:06.809306 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394bf1b9-d191-48df-ae60-5d3e4eca1bcf None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:06.809772 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394bf1b9-d191-48df-ae60-5d3e4eca1bcf None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:06.811260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 32/63] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:06:06.824385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600de1db-7866-49d8-9088-2e78c700961d None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:06.824953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600de1db-7866-49d8-9088-2e78c700961d None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:06.825396 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600de1db-7866-49d8-9088-2e78c700961d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:06.856438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-600de1db-7866-49d8-9088-2e78c700961d None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:06.858274 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-600de1db-7866-49d8-9088-2e78c700961d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:06.866589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:06.892154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 32/64] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:06.977090 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 33/65] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:06:06.978266 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2f7f62-2c17-4b95-bb31-9cacc48122e0 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:06.978628 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2f7f62-2c17-4b95-bb31-9cacc48122e0 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:06.978977 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2f7f62-2c17-4b95-bb31-9cacc48122e0 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:06.980355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 33/66] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:06:06.991388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8814ea91-f85f-4ca8-81f0-2571394e5e6e None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:06.991681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8814ea91-f85f-4ca8-81f0-2571394e5e6e None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:06.991963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8814ea91-f85f-4ca8-81f0-2571394e5e6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:07.017686 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-8814ea91-f85f-4ca8-81f0-2571394e5e6e None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:07.018903 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-8814ea91-f85f-4ca8-81f0-2571394e5e6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:07.075604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:07.108176 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 34/67] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:07.173702 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2f7f62-2c17-4b95-bb31-9cacc48122e0 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:07.310574 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d2d937-93c6-412b-88bb-3609de20c8cc None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:07.310892 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d2d937-93c6-412b-88bb-3609de20c8cc None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:07.311136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d2d937-93c6-412b-88bb-3609de20c8cc None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:07.312474 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 35/68] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:06:07.317464 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2f7f62-2c17-4b95-bb31-9cacc48122e0 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:07.318173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdeccc6-8bdc-40f5-bf81-b60b9278c414 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:07.318468 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdeccc6-8bdc-40f5-bf81-b60b9278c414 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:07.318741 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdeccc6-8bdc-40f5-bf81-b60b9278c414 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:07.351716 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-fcdeccc6-8bdc-40f5-bf81-b60b9278c414 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:07.353411 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-fcdeccc6-8bdc-40f5-bf81-b60b9278c414 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:07.363639 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e38cb4-059a-4ddc-9c4d-3381d42fca1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:07.364406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e38cb4-059a-4ddc-9c4d-3381d42fca1a None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:07.364689 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e38cb4-059a-4ddc-9c4d-3381d42fca1a None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:07.364961 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e38cb4-059a-4ddc-9c4d-3381d42fca1a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:07.378911 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e38cb4-059a-4ddc-9c4d-3381d42fca1a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:07.385353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-a0e38cb4-059a-4ddc-9c4d-3381d42fca1a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:07.385353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:07.385353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:07.385353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:07.385353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:07.385353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:07.385353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:07.385353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:07.385353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:07.385353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:07.385353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:07.385353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:07.385353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:07.385353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:07.385353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:07.385353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:07.386382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:07.386382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:07.386382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:07.386382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:07.386382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:07.386382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:07.386382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:07.386382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:07.386382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:07.386382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:07.386382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:07.386382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:07.386382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:07.386382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:07.386382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:07.386382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:07.387140 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:07.387140 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:07.387140 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:07.387140 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:07.387140 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:07.387140 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:07.387140 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:07.387140 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:07.387140 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:07.387140 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:07.387140 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:07.387140 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:06:07.387140 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 34/69] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:07 2022] GET /identity/v3/roles/member => generated 84 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:07.390541 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e38cb4-059a-4ddc-9c4d-3381d42fca1a None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:07.446343 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 36/70] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:07.447475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdeccc6-8bdc-40f5-bf81-b60b9278c414 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:07.466575 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:07.522499 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e38cb4-059a-4ddc-9c4d-3381d42fca1a None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:07.552589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdeccc6-8bdc-40f5-bf81-b60b9278c414 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:07.559030 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67b130d-06c2-452a-9ed1-5ff8636f8f17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:07.559857 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67b130d-06c2-452a-9ed1-5ff8636f8f17 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:07.560264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:07.561195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67b130d-06c2-452a-9ed1-5ff8636f8f17 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:07.561652 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67b130d-06c2-452a-9ed1-5ff8636f8f17 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:07.572869 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67b130d-06c2-452a-9ed1-5ff8636f8f17 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:07.578339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-c67b130d-06c2-452a-9ed1-5ff8636f8f17 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:06:07.578339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:07.578339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:07.578339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:07.578339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:07.578339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:07.578339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:07.578339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:07.578339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:07.578339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:07.578339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:07.578339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:07.578339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:07.578339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:07.578339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:07.578339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:07.579357 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:07.579357 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:07.579357 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:07.579357 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:07.579357 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:07.579357 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:07.579357 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:07.579357 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:07.579357 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:07.579357 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:07.579357 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:07.579357 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:07.579357 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:07.579357 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:07.579357 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:07.579357 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:07.580302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:07.580302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:07.580302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:07.580302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:07.580302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:07.580302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:07.580302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:07.580302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:07.580302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:07.580302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:07.580302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:06:07.580302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:06:07.582319 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 35/71] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:06:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:07.583899 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67b130d-06c2-452a-9ed1-5ff8636f8f17 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:07.591725 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b3d1e9-5b46-43b4-a34a-331e55b4912e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:07.592455 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b3d1e9-5b46-43b4-a34a-331e55b4912e None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:07.592738 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b3d1e9-5b46-43b4-a34a-331e55b4912e None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:07.593011 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b3d1e9-5b46-43b4-a34a-331e55b4912e None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:07.593681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b3d1e9-5b46-43b4-a34a-331e55b4912e None admin] RBAC: Adding query filter params (name=member) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:07.594022 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b3d1e9-5b46-43b4-a34a-331e55b4912e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:07.595807 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b3d1e9-5b46-43b4-a34a-331e55b4912e None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:07.609332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 37/72] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:07 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:07.609332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ddaff2-2025-439b-ac62-dc85dca0608a None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:07.609332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ddaff2-2025-439b-ac62-dc85dca0608a None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:07.609332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ddaff2-2025-439b-ac62-dc85dca0608a None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:07.609332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 38/73] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:07 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:07.609332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ddaff2-2025-439b-ac62-dc85dca0608a None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:07.698664 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67b130d-06c2-452a-9ed1-5ff8636f8f17 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:07.718588 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:07.720004 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ddaff2-2025-439b-ac62-dc85dca0608a None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:07.739258 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837178db-e950-4cc9-b16a-3570930f2e8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:07.740465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837178db-e950-4cc9-b16a-3570930f2e8e None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:07.740745 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837178db-e950-4cc9-b16a-3570930f2e8e None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:07.741021 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837178db-e950-4cc9-b16a-3570930f2e8e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:07.750761 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-837178db-e950-4cc9-b16a-3570930f2e8e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:07.753046 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7e7e41-0859-4fd8-8a23-482d7014cca6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:07.753647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7e7e41-0859-4fd8-8a23-482d7014cca6 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:07.753970 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7e7e41-0859-4fd8-8a23-482d7014cca6 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:07.754248 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7e7e41-0859-4fd8-8a23-482d7014cca6 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:07.755407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7e7e41-0859-4fd8-8a23-482d7014cca6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:07.755715 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7e7e41-0859-4fd8-8a23-482d7014cca6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:07.756363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-837178db-e950-4cc9-b16a-3570930f2e8e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:06:07.756363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:07.756363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:07.756363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:07.756363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:07.756363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:07.756363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:07.756363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:07.756363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:07.756363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:07.756363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:07.756363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:07.756363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:07.756363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:07.756363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:07.756363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:07.757344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:07.757344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:07.757344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:07.757344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:07.757344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:07.757344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:07.757344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:07.757344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:07.757344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:07.757344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:07.757344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:07.757344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:07.757344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:07.757344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:07.757344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:07.757344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:07.762545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:07.762545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:07.762545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:07.762545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:07.762545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:07.762545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:07.762545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:07.762545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:07.762545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:07.762545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:07.762545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:06:07.762545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:06:07.763274 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7e7e41-0859-4fd8-8a23-482d7014cca6 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:07.763353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 36/74] 10.222.0.26 () {56 vars in 1114 bytes} [Tue Sep 27 04:06:07 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:07.763353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837178db-e950-4cc9-b16a-3570930f2e8e None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:07.766790 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 39/75] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:07.768032 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1975a98a-c44e-4929-9968-fff204c2e1ed None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:07.768320 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1975a98a-c44e-4929-9968-fff204c2e1ed None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:07.768594 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1975a98a-c44e-4929-9968-fff204c2e1ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:07.786977 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-1975a98a-c44e-4929-9968-fff204c2e1ed None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:07.788155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-1975a98a-c44e-4929-9968-fff204c2e1ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:07.812353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:07.856173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 40/76] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:07.857073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1975a98a-c44e-4929-9968-fff204c2e1ed None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:07.857393 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837178db-e950-4cc9-b16a-3570930f2e8e None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:07.903587 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c1d934-ac89-42d8-8b63-a03c168b1373 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:07.904305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c1d934-ac89-42d8-8b63-a03c168b1373 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:07.904590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c1d934-ac89-42d8-8b63-a03c168b1373 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:07.904864 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c1d934-ac89-42d8-8b63-a03c168b1373 None admin] PATH_INFO: `/v3/users/fe29a891245a498fa7af105f29387508` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:07.918176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c1d934-ac89-42d8-8b63-a03c168b1373 None admin] RBAC: Authorizing `identity:get_user(user_id=fe29a891245a498fa7af105f29387508)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:07.919705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c1d934-ac89-42d8-8b63-a03c168b1373 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:07.925791 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 37/77] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:07 2022] GET /identity/v3/users/fe29a891245a498fa7af105f29387508 => generated 275 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:07.925791 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c1d934-ac89-42d8-8b63-a03c168b1373 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:07.953881 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:07.963266 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1975a98a-c44e-4929-9968-fff204c2e1ed None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:07.999690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a268bb92-c01f-48da-a18c-dbac4aaf8c33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:08.000328 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a268bb92-c01f-48da-a18c-dbac4aaf8c33 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:08.000618 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a268bb92-c01f-48da-a18c-dbac4aaf8c33 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:08.000890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a268bb92-c01f-48da-a18c-dbac4aaf8c33 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:08.002294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a268bb92-c01f-48da-a18c-dbac4aaf8c33 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:08.003715 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a268bb92-c01f-48da-a18c-dbac4aaf8c33 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:08.009980 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 41/78] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:06:07 2022] GET /identity/v3/domains/default => generated 199 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:08.009980 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a268bb92-c01f-48da-a18c-dbac4aaf8c33 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:08.025642 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c1d934-ac89-42d8-8b63-a03c168b1373 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:08.041390 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:08.060606 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2192466-1f9a-49ef-affe-9521e3c195ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:08.061214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2192466-1f9a-49ef-affe-9521e3c195ba None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:08.061495 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2192466-1f9a-49ef-affe-9521e3c195ba None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:08.061900 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2192466-1f9a-49ef-affe-9521e3c195ba None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:08.062628 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2192466-1f9a-49ef-affe-9521e3c195ba None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:08.062877 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2192466-1f9a-49ef-affe-9521e3c195ba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:08.064349 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2192466-1f9a-49ef-affe-9521e3c195ba None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:08.072655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 38/79] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:06:07 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:08.073441 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2192466-1f9a-49ef-affe-9521e3c195ba None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:08.125055 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a268bb92-c01f-48da-a18c-dbac4aaf8c33 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:08.160919 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77cfa62-a7a9-4392-8503-03a49e5eebee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:08.161688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77cfa62-a7a9-4392-8503-03a49e5eebee None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:08.162087 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77cfa62-a7a9-4392-8503-03a49e5eebee None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:08.162431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77cfa62-a7a9-4392-8503-03a49e5eebee None admin] PATH_INFO: `/v3/users/6190ca8b8acd4e99b228f0851e2ed5af` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:08.163586 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77cfa62-a7a9-4392-8503-03a49e5eebee None admin] RBAC: Authorizing `identity:get_user(user_id=6190ca8b8acd4e99b228f0851e2ed5af)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:08.163705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:08.165034 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77cfa62-a7a9-4392-8503-03a49e5eebee None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:08.166713 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 42/80] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:08 2022] GET /identity/v3/users/6190ca8b8acd4e99b228f0851e2ed5af => generated 247 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:08.167459 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77cfa62-a7a9-4392-8503-03a49e5eebee None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:08.176535 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2192466-1f9a-49ef-affe-9521e3c195ba None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:08.209650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64be6d1-33c2-421e-8bde-af45ee061843 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:08.210275 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64be6d1-33c2-421e-8bde-af45ee061843 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:08.210554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64be6d1-33c2-421e-8bde-af45ee061843 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:08.210830 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64be6d1-33c2-421e-8bde-af45ee061843 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:08.258565 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:08.266721 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77cfa62-a7a9-4392-8503-03a49e5eebee None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:08.284213 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64be6d1-33c2-421e-8bde-af45ee061843 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:08.285681 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64be6d1-33c2-421e-8bde-af45ee061843 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:08.304082 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9d89a9-0dc7-4bb3-9e19-4c89efd914af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:08.304771 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9d89a9-0dc7-4bb3-9e19-4c89efd914af None admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:08.305065 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9d89a9-0dc7-4bb3-9e19-4c89efd914af None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:08.305332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9d89a9-0dc7-4bb3-9e19-4c89efd914af None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:08.306029 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9d89a9-0dc7-4bb3-9e19-4c89efd914af None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:08.307472 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9d89a9-0dc7-4bb3-9e19-4c89efd914af None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:08.335292 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 43/81] 10.222.0.26 () {60 vars in 1132 bytes} [Tue Sep 27 04:06:08 2022] POST /identity/v3/users => generated 289 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:06:08.336045 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9d89a9-0dc7-4bb3-9e19-4c89efd914af None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:08.380023 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:08.397238 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 39/82] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:06:08.397238 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64be6d1-33c2-421e-8bde-af45ee061843 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:08.461540 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9d89a9-0dc7-4bb3-9e19-4c89efd914af None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:08.502720 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdff78e9-dd54-4e3d-bc64-88af79239907 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:08.503371 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdff78e9-dd54-4e3d-bc64-88af79239907 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:08.503650 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdff78e9-dd54-4e3d-bc64-88af79239907 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:08.503926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdff78e9-dd54-4e3d-bc64-88af79239907 None admin] PATH_INFO: `/v3/users/fe29a891245a498fa7af105f29387508` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:08.505126 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdff78e9-dd54-4e3d-bc64-88af79239907 None admin] RBAC: Authorizing `identity:get_user(user_id=fe29a891245a498fa7af105f29387508)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:08.506600 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdff78e9-dd54-4e3d-bc64-88af79239907 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:08.508719 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 44/83] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:08 2022] GET /identity/v3/users/fe29a891245a498fa7af105f29387508 => generated 275 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:08.508937 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdff78e9-dd54-4e3d-bc64-88af79239907 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:08.523795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64be6d1-33c2-421e-8bde-af45ee061843 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:08.534602 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:08.563629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7cc777-38cb-489b-978f-673d1a1c047f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:08.564348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7cc777-38cb-489b-978f-673d1a1c047f None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:08.564691 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7cc777-38cb-489b-978f-673d1a1c047f None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:08.565029 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7cc777-38cb-489b-978f-673d1a1c047f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:08.578447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7cc777-38cb-489b-978f-673d1a1c047f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:08.582592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-8b7cc777-38cb-489b-978f-673d1a1c047f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:08.582592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:08.582592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:08.582592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:08.582592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:08.582592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:08.582592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:08.582592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:08.582592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:08.582592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:08.582592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:08.582592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:08.582592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:08.582592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:08.582592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:08.582592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:08.584121 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:08.584121 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:08.584121 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:08.584121 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:08.584121 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:08.584121 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:08.584121 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:08.584121 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:08.584121 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:08.584121 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:08.584121 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:08.584121 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:08.584121 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:08.584121 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:08.584121 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:08.584121 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:08.585131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:08.585131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:08.585131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:08.585131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:08.585131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:08.585131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:08.585131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:08.585131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:08.585131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:08.585131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:08.585131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:08.585131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:06:08.585131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 40/84] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:08 2022] GET /identity/v3/roles/member => generated 84 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:08.586332 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7cc777-38cb-489b-978f-673d1a1c047f None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:08.651699 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdff78e9-dd54-4e3d-bc64-88af79239907 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:08.652964 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:08.695897 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d8f0db-142c-4130-9c5c-78f2e3fd17f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:08.697922 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d8f0db-142c-4130-9c5c-78f2e3fd17f7 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:08.697922 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d8f0db-142c-4130-9c5c-78f2e3fd17f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:08.697922 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d8f0db-142c-4130-9c5c-78f2e3fd17f7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:08.705821 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7cc777-38cb-489b-978f-673d1a1c047f None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:08.749560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785b7df2-a531-4134-979b-398b027d35a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:08.750929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785b7df2-a531-4134-979b-398b027d35a6 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:08.751612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785b7df2-a531-4134-979b-398b027d35a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:08.752024 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785b7df2-a531-4134-979b-398b027d35a6 None admin] PATH_INFO: `/v3/projects/146fd1189c56413c9d17cd16d1a974d3` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:08.753301 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785b7df2-a531-4134-979b-398b027d35a6 None admin] RBAC: Authorizing `identity:get_project(project_id=146fd1189c56413c9d17cd16d1a974d3)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:08.754933 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785b7df2-a531-4134-979b-398b027d35a6 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:08.756689 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 41/85] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:08 2022] GET /identity/v3/projects/146fd1189c56413c9d17cd16d1a974d3 => generated 298 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:08.757576 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785b7df2-a531-4134-979b-398b027d35a6 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:08.785015 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d8f0db-142c-4130-9c5c-78f2e3fd17f7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:08.787193 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d8f0db-142c-4130-9c5c-78f2e3fd17f7 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:08.812020 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:08.864205 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785b7df2-a531-4134-979b-398b027d35a6 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:08.891118 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:08.910647 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1526d8ca-1b4a-4955-b14f-5f46a08a0a29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:08.911507 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1526d8ca-1b4a-4955-b14f-5f46a08a0a29 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:08.911790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1526d8ca-1b4a-4955-b14f-5f46a08a0a29 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:08.912068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1526d8ca-1b4a-4955-b14f-5f46a08a0a29 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:08.919507 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 45/86] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:06:08.920186 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d8f0db-142c-4130-9c5c-78f2e3fd17f7 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:09.002051 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1526d8ca-1b4a-4955-b14f-5f46a08a0a29 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:09.003405 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1526d8ca-1b4a-4955-b14f-5f46a08a0a29 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:09.042679 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d8f0db-142c-4130-9c5c-78f2e3fd17f7 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:09.069882 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:09.088400 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3cd69a-d41e-4f37-9aac-89ed23d8552c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:09.089043 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3cd69a-d41e-4f37-9aac-89ed23d8552c None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:09.089334 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3cd69a-d41e-4f37-9aac-89ed23d8552c None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:09.089604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3cd69a-d41e-4f37-9aac-89ed23d8552c None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:09.090287 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3cd69a-d41e-4f37-9aac-89ed23d8552c None admin] RBAC: Adding query filter params (name=member) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:09.090590 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3cd69a-d41e-4f37-9aac-89ed23d8552c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:09.091958 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3cd69a-d41e-4f37-9aac-89ed23d8552c None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:09.100442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 46/87] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:08 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:09.101243 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3cd69a-d41e-4f37-9aac-89ed23d8552c None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:09.133251 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 42/88] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:06:09.139111 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1526d8ca-1b4a-4955-b14f-5f46a08a0a29 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:09.161919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:09.219042 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3cd69a-d41e-4f37-9aac-89ed23d8552c None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:09.267377 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1526d8ca-1b4a-4955-b14f-5f46a08a0a29 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:09.268665 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf16c7b-b6da-4f86-8c3f-727f82e758c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:09.269664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf16c7b-b6da-4f86-8c3f-727f82e758c2 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:09.270029 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf16c7b-b6da-4f86-8c3f-727f82e758c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:09.271881 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf16c7b-b6da-4f86-8c3f-727f82e758c2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:09.272705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf16c7b-b6da-4f86-8c3f-727f82e758c2 None admin] RBAC: Adding query filter params (role.id=e3ce5a3697ec4292a446366d3caadeea, scope.project.id=146fd1189c56413c9d17cd16d1a974d3, user.id=fe29a891245a498fa7af105f29387508) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:09.273089 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf16c7b-b6da-4f86-8c3f-727f82e758c2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:09.274644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf16c7b-b6da-4f86-8c3f-727f82e758c2 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:09.292357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 47/89] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:06:09 2022] GET /identity/v3/role_assignments?user.id=fe29a891245a498fa7af105f29387508&scope.project.id=146fd1189c56413c9d17cd16d1a974d3&role.id=e3ce5a3697ec4292a446366d3caadeea => generated 259 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:09.292357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf16c7b-b6da-4f86-8c3f-727f82e758c2 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:09.336971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb08faa-daf6-4f1b-9ff4-7cf3fb3f2db6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:09.337963 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb08faa-daf6-4f1b-9ff4-7cf3fb3f2db6 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:09.338391 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb08faa-daf6-4f1b-9ff4-7cf3fb3f2db6 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:09.338807 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb08faa-daf6-4f1b-9ff4-7cf3fb3f2db6 None admin] PATH_INFO: `/v3/projects/146fd1189c56413c9d17cd16d1a974d3` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:09.342496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb08faa-daf6-4f1b-9ff4-7cf3fb3f2db6 None admin] RBAC: Authorizing `identity:get_project(project_id=146fd1189c56413c9d17cd16d1a974d3)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:09.344271 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb08faa-daf6-4f1b-9ff4-7cf3fb3f2db6 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:09.346562 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 43/90] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:09 2022] GET /identity/v3/projects/146fd1189c56413c9d17cd16d1a974d3 => generated 298 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:09.347542 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb08faa-daf6-4f1b-9ff4-7cf3fb3f2db6 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:09.348765 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:09.420773 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf16c7b-b6da-4f86-8c3f-727f82e758c2 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:09.464512 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af275fb2-6ce4-47b9-b5da-ac383106f5a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:09.465669 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af275fb2-6ce4-47b9-b5da-ac383106f5a8 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:09.466166 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af275fb2-6ce4-47b9-b5da-ac383106f5a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:09.466757 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af275fb2-6ce4-47b9-b5da-ac383106f5a8 None admin] PATH_INFO: `/v3/users/fe29a891245a498fa7af105f29387508` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:09.468251 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af275fb2-6ce4-47b9-b5da-ac383106f5a8 None admin] RBAC: Authorizing `identity:get_user(user_id=fe29a891245a498fa7af105f29387508)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:09.469963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af275fb2-6ce4-47b9-b5da-ac383106f5a8 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:09.471833 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 48/91] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:09 2022] GET /identity/v3/users/fe29a891245a498fa7af105f29387508 => generated 275 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:09.472813 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af275fb2-6ce4-47b9-b5da-ac383106f5a8 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:09.552782 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:09.582040 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb08faa-daf6-4f1b-9ff4-7cf3fb3f2db6 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:09.620947 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af275fb2-6ce4-47b9-b5da-ac383106f5a8 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:09.641236 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52627cc-2e40-4dcd-87a5-081ba5151370 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:09.641950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52627cc-2e40-4dcd-87a5-081ba5151370 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:09.642229 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52627cc-2e40-4dcd-87a5-081ba5151370 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:09.642501 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52627cc-2e40-4dcd-87a5-081ba5151370 None admin] PATH_INFO: `/v3/projects/146fd1189c56413c9d17cd16d1a974d3` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:09.643680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52627cc-2e40-4dcd-87a5-081ba5151370 None admin] RBAC: Authorizing `identity:get_project(project_id=146fd1189c56413c9d17cd16d1a974d3)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:09.645121 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52627cc-2e40-4dcd-87a5-081ba5151370 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:09.653930 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 44/92] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:09 2022] GET /identity/v3/projects/146fd1189c56413c9d17cd16d1a974d3 => generated 298 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:09.654881 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52627cc-2e40-4dcd-87a5-081ba5151370 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:09.660156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:09.683280 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4746aa-7f14-418d-8228-ef4840b52d8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:09.683806 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4746aa-7f14-418d-8228-ef4840b52d8e None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:09.684086 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4746aa-7f14-418d-8228-ef4840b52d8e None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:09.684360 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4746aa-7f14-418d-8228-ef4840b52d8e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:09.685091 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4746aa-7f14-418d-8228-ef4840b52d8e None admin] RBAC: Adding query filter params (role.id=e6a31562a18649f0809202c36a8e18ef, scope.project.id=146fd1189c56413c9d17cd16d1a974d3, user.id=6190ca8b8acd4e99b228f0851e2ed5af) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:09.685398 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4746aa-7f14-418d-8228-ef4840b52d8e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:09.686859 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4746aa-7f14-418d-8228-ef4840b52d8e None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:09.701988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 49/93] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:06:09 2022] GET /identity/v3/role_assignments?user.id=6190ca8b8acd4e99b228f0851e2ed5af&scope.project.id=146fd1189c56413c9d17cd16d1a974d3&role.id=e6a31562a18649f0809202c36a8e18ef => generated 259 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:09.702646 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4746aa-7f14-418d-8228-ef4840b52d8e None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:09.857891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4746aa-7f14-418d-8228-ef4840b52d8e None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:09.903923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52627cc-2e40-4dcd-87a5-081ba5151370 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:09.944219 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed369fb-33f4-4ee8-9109-d0b98395cb8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:09.944759 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed369fb-33f4-4ee8-9109-d0b98395cb8a None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:09.945037 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed369fb-33f4-4ee8-9109-d0b98395cb8a None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:09.945303 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed369fb-33f4-4ee8-9109-d0b98395cb8a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:09.946065 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed369fb-33f4-4ee8-9109-d0b98395cb8a None admin] RBAC: Adding query filter params (role.id=2d9f391485604d0b952b7c844c3a3266, scope.project.id=146fd1189c56413c9d17cd16d1a974d3, user.id=fe29a891245a498fa7af105f29387508) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:09.946376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed369fb-33f4-4ee8-9109-d0b98395cb8a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:09.947796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed369fb-33f4-4ee8-9109-d0b98395cb8a None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:09.962392 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 50/94] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:06:09 2022] GET /identity/v3/role_assignments?user.id=fe29a891245a498fa7af105f29387508&scope.project.id=146fd1189c56413c9d17cd16d1a974d3&role.id=2d9f391485604d0b952b7c844c3a3266 => generated 259 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:09.979199 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce8e83e-81dc-4df2-b8ff-f0830d5da64c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:09.979815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce8e83e-81dc-4df2-b8ff-f0830d5da64c None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:09.980092 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce8e83e-81dc-4df2-b8ff-f0830d5da64c None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:09.980362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce8e83e-81dc-4df2-b8ff-f0830d5da64c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:10.026997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:10.089727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce8e83e-81dc-4df2-b8ff-f0830d5da64c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:10.091247 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce8e83e-81dc-4df2-b8ff-f0830d5da64c None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:10.227584 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 45/95] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 573 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:06:10.233441 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed369fb-33f4-4ee8-9109-d0b98395cb8a None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:10.307352 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:10.377888 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed369fb-33f4-4ee8-9109-d0b98395cb8a None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:10.426585 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b6c8d6-4672-4deb-b582-d4487c8833d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:10.427294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b6c8d6-4672-4deb-b582-d4487c8833d8 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:10.427576 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b6c8d6-4672-4deb-b582-d4487c8833d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:10.427853 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b6c8d6-4672-4deb-b582-d4487c8833d8 None admin] PATH_INFO: `/v3/projects/4a259a88e57d4f71bf57358d1fce3cc3` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:10.435340 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b6c8d6-4672-4deb-b582-d4487c8833d8 None admin] RBAC: Authorizing `identity:get_project(project_id=4a259a88e57d4f71bf57358d1fce3cc3)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:10.436837 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b6c8d6-4672-4deb-b582-d4487c8833d8 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:10.439012 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 51/96] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:10 2022] GET /identity/v3/projects/4a259a88e57d4f71bf57358d1fce3cc3 => generated 312 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:10.445905 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce8e83e-81dc-4df2-b8ff-f0830d5da64c None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:10.620302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:10.714775 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce8e83e-81dc-4df2-b8ff-f0830d5da64c None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:10.794485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f84789-b232-4c47-bb84-7ea422c3988b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:10.795442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f84789-b232-4c47-bb84-7ea422c3988b None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:10.795863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f84789-b232-4c47-bb84-7ea422c3988b None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:10.796280 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f84789-b232-4c47-bb84-7ea422c3988b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:10.797153 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f84789-b232-4c47-bb84-7ea422c3988b None admin] RBAC: Adding query filter params (role.id=e3ce5a3697ec4292a446366d3caadeea, scope.project.id=4a259a88e57d4f71bf57358d1fce3cc3, user.id=fe29a891245a498fa7af105f29387508) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:10.797600 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f84789-b232-4c47-bb84-7ea422c3988b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:10.799221 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f84789-b232-4c47-bb84-7ea422c3988b None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:10.805874 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 46/97] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:06:10 2022] GET /identity/v3/role_assignments?user.id=fe29a891245a498fa7af105f29387508&scope.project.id=4a259a88e57d4f71bf57358d1fce3cc3&role.id=e3ce5a3697ec4292a446366d3caadeea => generated 259 bytes in 361 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:11.036354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34600abc-2919-4ad7-b6fe-f3ae10e84988 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:11.037214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34600abc-2919-4ad7-b6fe-f3ae10e84988 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:11.037634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34600abc-2919-4ad7-b6fe-f3ae10e84988 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:11.040576 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 52/98] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:11.045075 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7411c65b-cb36-4fed-9268-49a2047bdc0d None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:11.045880 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7411c65b-cb36-4fed-9268-49a2047bdc0d None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:11.046297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7411c65b-cb36-4fed-9268-49a2047bdc0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:11.083165 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-7411c65b-cb36-4fed-9268-49a2047bdc0d None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:11.084640 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-7411c65b-cb36-4fed-9268-49a2047bdc0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:11.179329 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 47/99] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:11.551369 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34600abc-2919-4ad7-b6fe-f3ae10e84988 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:11.561986 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d001dc18-4a34-4253-94e2-e6b3595a94c6 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:11.564264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d001dc18-4a34-4253-94e2-e6b3595a94c6 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:11.564739 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d001dc18-4a34-4253-94e2-e6b3595a94c6 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:11.566326 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 48/100] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:11 2022] GET /identity => generated 270 bytes in 13 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:11.571939 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ba021f-fdf3-4a49-a585-ada44e2bf6c4 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:11.572455 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ba021f-fdf3-4a49-a585-ada44e2bf6c4 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:11.572917 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ba021f-fdf3-4a49-a585-ada44e2bf6c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:11.605367 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-e5ba021f-fdf3-4a49-a585-ada44e2bf6c4 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:11.607574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-e5ba021f-fdf3-4a49-a585-ada44e2bf6c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:11.697899 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:11.736402 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 49/101] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:11.738337 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34600abc-2919-4ad7-b6fe-f3ae10e84988 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:11.753608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:11.779693 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88e6b49-595b-4aa3-921b-64dac1ff62b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:11.780519 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88e6b49-595b-4aa3-921b-64dac1ff62b9 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:11.780958 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88e6b49-595b-4aa3-921b-64dac1ff62b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:11.781367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88e6b49-595b-4aa3-921b-64dac1ff62b9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:11.782632 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88e6b49-595b-4aa3-921b-64dac1ff62b9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:11.784197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88e6b49-595b-4aa3-921b-64dac1ff62b9 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:11.785987 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 53/102] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:06:11 2022] GET /identity/v3/domains/default => generated 199 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:11.792264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ba021f-fdf3-4a49-a585-ada44e2bf6c4 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:11.901946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ba021f-fdf3-4a49-a585-ada44e2bf6c4 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:11.943566 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5f2ad5-6ec5-468d-a33b-0abf6c97fc20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:11.944483 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5f2ad5-6ec5-468d-a33b-0abf6c97fc20 None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:11.944897 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5f2ad5-6ec5-468d-a33b-0abf6c97fc20 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:11.945304 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5f2ad5-6ec5-468d-a33b-0abf6c97fc20 None admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:11.946129 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5f2ad5-6ec5-468d-a33b-0abf6c97fc20 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:11.947715 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5f2ad5-6ec5-468d-a33b-0abf6c97fc20 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:11.988230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 50/103] 10.222.0.26 () {60 vars in 1141 bytes} [Tue Sep 27 04:06:11 2022] POST /identity/v3/projects => generated 302 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:06:12.290595 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88e6b49-595b-4aa3-921b-64dac1ff62b9 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:12.425794 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88e6b49-595b-4aa3-921b-64dac1ff62b9 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:12.460311 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382113c8-2e24-4c75-bf50-be7cca2d4e70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:12.460939 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382113c8-2e24-4c75-bf50-be7cca2d4e70 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:12.461217 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382113c8-2e24-4c75-bf50-be7cca2d4e70 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:12.461489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382113c8-2e24-4c75-bf50-be7cca2d4e70 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:12.473290 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-382113c8-2e24-4c75-bf50-be7cca2d4e70 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:12.479388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-382113c8-2e24-4c75-bf50-be7cca2d4e70 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 04:06:12.479388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:12.479388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:12.479388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:12.479388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:12.479388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:12.479388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:12.479388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:12.479388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:12.479388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:12.479388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:12.479388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:12.479388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:12.479388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:12.479388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:12.479388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:12.480214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:12.480214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:12.480214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:12.480214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:12.480214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:12.480214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:12.480214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:12.480214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:12.480214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:12.480214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:12.480214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:12.480214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:12.480214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:12.480214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:12.480214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:12.480214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:12.480979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:12.480979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:12.480979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:12.480979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:12.480979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:12.480979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:12.480979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:12.480979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:12.480979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:12.480979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:12.480979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 04:06:12.480979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:06:12.480979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 54/104] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:12 2022] GET /identity/v3/roles/reader => generated 84 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:12.486832 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5f2ad5-6ec5-468d-a33b-0abf6c97fc20 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:12.520546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:12.609065 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5f2ad5-6ec5-468d-a33b-0abf6c97fc20 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:12.653307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180dd9ce-cc7d-4539-aec3-95d49ee407c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:12.654312 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180dd9ce-cc7d-4539-aec3-95d49ee407c4 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:12.654797 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180dd9ce-cc7d-4539-aec3-95d49ee407c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:12.655435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180dd9ce-cc7d-4539-aec3-95d49ee407c4 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:12.656331 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-180dd9ce-cc7d-4539-aec3-95d49ee407c4 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:12.656856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-180dd9ce-cc7d-4539-aec3-95d49ee407c4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:12.658467 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-180dd9ce-cc7d-4539-aec3-95d49ee407c4 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:12.667740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 51/105] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:12 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:12.673568 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382113c8-2e24-4c75-bf50-be7cca2d4e70 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:12.693281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f845eccf-026c-4a41-b4f6-8e536a22d0b8 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:12.693948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f845eccf-026c-4a41-b4f6-8e536a22d0b8 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:12.694427 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f845eccf-026c-4a41-b4f6-8e536a22d0b8 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:12.695966 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 52/106] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:06:12.702751 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a32cc4e-2f0e-4ab1-8e1e-126d8fd06979 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:12.703044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a32cc4e-2f0e-4ab1-8e1e-126d8fd06979 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:12.703327 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a32cc4e-2f0e-4ab1-8e1e-126d8fd06979 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:12.722571 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-8a32cc4e-2f0e-4ab1-8e1e-126d8fd06979 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:12.724308 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-8a32cc4e-2f0e-4ab1-8e1e-126d8fd06979 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:12.738902 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:12.853892 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 53/107] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:06:12.867766 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382113c8-2e24-4c75-bf50-be7cca2d4e70 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:12.943704 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcd1292-24d8-4919-b17a-9eb44f33150d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:12.944364 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcd1292-24d8-4919-b17a-9eb44f33150d None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:12.944653 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcd1292-24d8-4919-b17a-9eb44f33150d None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:12.944929 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcd1292-24d8-4919-b17a-9eb44f33150d None admin] PATH_INFO: `/v3/users/71826307b6884fe78abc8b76724601d4` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:12.969784 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dcd1292-24d8-4919-b17a-9eb44f33150d None admin] RBAC: Authorizing `identity:get_user(user_id=71826307b6884fe78abc8b76724601d4)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:12.971257 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dcd1292-24d8-4919-b17a-9eb44f33150d None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:12.972957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 55/108] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:12 2022] GET /identity/v3/users/71826307b6884fe78abc8b76724601d4 => generated 289 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:12.979277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a32cc4e-2f0e-4ab1-8e1e-126d8fd06979 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:13.029666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:13.125581 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a32cc4e-2f0e-4ab1-8e1e-126d8fd06979 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:13.159313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c848c4-f0a2-4dd7-9ca1-d5dad6c2eece None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:13.159629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c848c4-f0a2-4dd7-9ca1-d5dad6c2eece None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:13.159873 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c848c4-f0a2-4dd7-9ca1-d5dad6c2eece None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:13.161918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 56/109] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:13.164543 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d71f0b6-30ec-4fe4-a133-42cd2d9e5c32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:13.165160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d71f0b6-30ec-4fe4-a133-42cd2d9e5c32 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:13.165439 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d71f0b6-30ec-4fe4-a133-42cd2d9e5c32 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:13.165802 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d71f0b6-30ec-4fe4-a133-42cd2d9e5c32 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:13.166922 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdf96ca-9c98-46e6-908d-9c81ebebacde None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:13.167211 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdf96ca-9c98-46e6-908d-9c81ebebacde None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:13.167496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdf96ca-9c98-46e6-908d-9c81ebebacde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:13.201927 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-8bdf96ca-9c98-46e6-908d-9c81ebebacde None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:13.203136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-8bdf96ca-9c98-46e6-908d-9c81ebebacde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:13.277409 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d71f0b6-30ec-4fe4-a133-42cd2d9e5c32 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:13.278907 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d71f0b6-30ec-4fe4-a133-42cd2d9e5c32 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:13.312144 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:13.374284 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 57/110] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:13.374963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bdf96ca-9c98-46e6-908d-9c81ebebacde None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:13.394193 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 54/111] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:06:13.400539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d71f0b6-30ec-4fe4-a133-42cd2d9e5c32 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:13.468619 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:13.542828 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d71f0b6-30ec-4fe4-a133-42cd2d9e5c32 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:13.586911 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:13.615055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12af9e40-3b1f-48d7-ae29-5c17d616d017 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:13.615715 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12af9e40-3b1f-48d7-ae29-5c17d616d017 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:13.615997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12af9e40-3b1f-48d7-ae29-5c17d616d017 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:13.616535 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12af9e40-3b1f-48d7-ae29-5c17d616d017 None admin] PATH_INFO: `/v3/projects/146fd1189c56413c9d17cd16d1a974d3` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:13.628148 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12af9e40-3b1f-48d7-ae29-5c17d616d017 None admin] RBAC: Authorizing `identity:get_project(project_id=146fd1189c56413c9d17cd16d1a974d3)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:13.628148 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12af9e40-3b1f-48d7-ae29-5c17d616d017 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:13.628148 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 55/112] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:13 2022] GET /identity/v3/projects/146fd1189c56413c9d17cd16d1a974d3 => generated 298 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:13.636862 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12af9e40-3b1f-48d7-ae29-5c17d616d017 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:13.685868 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bdf96ca-9c98-46e6-908d-9c81ebebacde None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:13.735404 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17877263-44c9-4d72-8ae1-26079af1180e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:13.736040 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17877263-44c9-4d72-8ae1-26079af1180e None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:13.736322 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17877263-44c9-4d72-8ae1-26079af1180e None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:13.736723 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17877263-44c9-4d72-8ae1-26079af1180e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:13.748355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17877263-44c9-4d72-8ae1-26079af1180e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:13.754884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-17877263-44c9-4d72-8ae1-26079af1180e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:13.754884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:13.754884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:13.754884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:13.754884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:13.754884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:13.754884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:13.754884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:13.754884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:13.754884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:13.754884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:13.754884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:13.754884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:13.754884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:13.754884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:13.754884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:13.756244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:13.756244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:13.756244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:13.756244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:13.756244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:13.756244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:13.756244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:13.756244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:13.756244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:13.756244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:13.756244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:13.756244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:13.756244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:13.756244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:13.756244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:13.756244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:13.756988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:13.756988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:13.756988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:13.756988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:13.756988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:13.756988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:13.756988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:13.756988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:13.756988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:13.756988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:13.756988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:13.756988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:06:13.761274 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 58/113] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:13 2022] GET /identity/v3/roles/member => generated 84 bytes in 382 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:13.762126 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17877263-44c9-4d72-8ae1-26079af1180e None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:13.803486 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12af9e40-3b1f-48d7-ae29-5c17d616d017 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:13.843109 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c738a62a-a68f-4a43-9ae8-97bd4da0d5ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:13.844091 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c738a62a-a68f-4a43-9ae8-97bd4da0d5ae None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:13.844584 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c738a62a-a68f-4a43-9ae8-97bd4da0d5ae None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:13.845046 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c738a62a-a68f-4a43-9ae8-97bd4da0d5ae None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:13.845975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c738a62a-a68f-4a43-9ae8-97bd4da0d5ae None admin] RBAC: Adding query filter params (role.id=9b729a0f423e47f7bc562c000780ef7b, scope.project.id=146fd1189c56413c9d17cd16d1a974d3, user.id=71826307b6884fe78abc8b76724601d4) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:13.846476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c738a62a-a68f-4a43-9ae8-97bd4da0d5ae None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:13.848111 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c738a62a-a68f-4a43-9ae8-97bd4da0d5ae None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:13.854120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 56/114] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:06:13 2022] GET /identity/v3/role_assignments?user.id=71826307b6884fe78abc8b76724601d4&scope.project.id=146fd1189c56413c9d17cd16d1a974d3&role.id=9b729a0f423e47f7bc562c000780ef7b => generated 259 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:13.861703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654e1771-0366-48ea-9115-232cfaa71081 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:13.862651 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654e1771-0366-48ea-9115-232cfaa71081 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:13.863168 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654e1771-0366-48ea-9115-232cfaa71081 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:13.865667 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 57/115] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:13 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:13.866757 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654e1771-0366-48ea-9115-232cfaa71081 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:13.872960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:13.891389 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17877263-44c9-4d72-8ae1-26079af1180e None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:13.933918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60714de-6146-4387-81e0-6360396d4d4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:13.934566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60714de-6146-4387-81e0-6360396d4d4d None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:13.934845 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60714de-6146-4387-81e0-6360396d4d4d None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:13.935118 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60714de-6146-4387-81e0-6360396d4d4d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:13.948750 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60714de-6146-4387-81e0-6360396d4d4d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:13.958355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-e60714de-6146-4387-81e0-6360396d4d4d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:06:13.958355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:13.958355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:13.958355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:13.958355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:13.958355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:13.958355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:13.958355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:13.958355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:13.958355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:13.958355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:13.958355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:13.958355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:13.958355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:13.958355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:13.958355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:13.959249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:13.959249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:13.959249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:13.959249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:13.959249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:13.959249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:13.959249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:13.959249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:13.959249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:13.959249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:13.959249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:13.959249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:13.959249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:13.959249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:13.959249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:13.959249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:13.959981 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:13.959981 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:13.959981 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:13.959981 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:13.959981 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:13.959981 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:13.959981 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:13.959981 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:13.959981 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:13.959981 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:13.959981 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:06:13.959981 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:06:13.959981 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 59/116] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:06:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:13.961354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97625fdf-59b0-4a91-992f-60316c27e2bc None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:13.961686 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97625fdf-59b0-4a91-992f-60316c27e2bc None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:13.962065 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97625fdf-59b0-4a91-992f-60316c27e2bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:13.983777 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-97625fdf-59b0-4a91-992f-60316c27e2bc None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:13.985052 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-97625fdf-59b0-4a91-992f-60316c27e2bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:13.995414 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:14.052425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654e1771-0366-48ea-9115-232cfaa71081 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:14.073556 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 60/117] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:14.074904 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97625fdf-59b0-4a91-992f-60316c27e2bc None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:14.112527 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e1bc33-43f2-47ec-ab0c-416b489f69c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:14.113152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e1bc33-43f2-47ec-ab0c-416b489f69c0 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:14.113433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e1bc33-43f2-47ec-ab0c-416b489f69c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:14.113709 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e1bc33-43f2-47ec-ab0c-416b489f69c0 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:14.114947 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e1bc33-43f2-47ec-ab0c-416b489f69c0 None admin] RBAC: Adding query filter params (name=member) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:14.115252 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e1bc33-43f2-47ec-ab0c-416b489f69c0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:14.116634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e1bc33-43f2-47ec-ab0c-416b489f69c0 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:14.131198 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 58/118] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:13 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:14.131198 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b04a12-23eb-492e-a26e-2638c050c8d2 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:14.131198 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b04a12-23eb-492e-a26e-2638c050c8d2 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:14.131198 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b04a12-23eb-492e-a26e-2638c050c8d2 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:14.131198 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 59/119] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:14.135874 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b04a12-23eb-492e-a26e-2638c050c8d2 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:14.194612 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:14.241654 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97625fdf-59b0-4a91-992f-60316c27e2bc None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:14.304769 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b04a12-23eb-492e-a26e-2638c050c8d2 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:14.318948 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03610e4c-845d-43b4-9e2f-8f79311b6bd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:14.319865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03610e4c-845d-43b4-9e2f-8f79311b6bd2 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:14.320336 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03610e4c-845d-43b4-9e2f-8f79311b6bd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:14.320796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03610e4c-845d-43b4-9e2f-8f79311b6bd2 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:14.321652 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03610e4c-845d-43b4-9e2f-8f79311b6bd2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:14.322168 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03610e4c-845d-43b4-9e2f-8f79311b6bd2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:14.325796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03610e4c-845d-43b4-9e2f-8f79311b6bd2 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:14.343504 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 61/120] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:14.347367 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:14.353077 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ac1bed-956f-44e4-8182-9c4251199fc1 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:14.353646 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ac1bed-956f-44e4-8182-9c4251199fc1 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:14.354174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ac1bed-956f-44e4-8182-9c4251199fc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:14.378361 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-70ac1bed-956f-44e4-8182-9c4251199fc1 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:14.380277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8a9119-af64-40de-8a1f-1753a89deeb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:14.380556 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-70ac1bed-956f-44e4-8182-9c4251199fc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:14.380958 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8a9119-af64-40de-8a1f-1753a89deeb4 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:14.381238 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8a9119-af64-40de-8a1f-1753a89deeb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:14.381559 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8a9119-af64-40de-8a1f-1753a89deeb4 None admin] PATH_INFO: `/v3/users/fe29a891245a498fa7af105f29387508` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:14.382807 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8a9119-af64-40de-8a1f-1753a89deeb4 None admin] RBAC: Authorizing `identity:get_user(user_id=fe29a891245a498fa7af105f29387508)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:14.385495 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8a9119-af64-40de-8a1f-1753a89deeb4 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:14.391625 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 60/121] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:14 2022] GET /identity/v3/users/fe29a891245a498fa7af105f29387508 => generated 275 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:14.391625 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8a9119-af64-40de-8a1f-1753a89deeb4 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:14.464939 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 62/122] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:14.466338 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ac1bed-956f-44e4-8182-9c4251199fc1 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:14.521698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8a9119-af64-40de-8a1f-1753a89deeb4 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:14.538996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:14.572725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5803d0af-4a85-46e7-a24e-631747da5cbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:14.574203 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5803d0af-4a85-46e7-a24e-631747da5cbb None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:14.575769 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5803d0af-4a85-46e7-a24e-631747da5cbb None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:14.577000 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5803d0af-4a85-46e7-a24e-631747da5cbb None admin] PATH_INFO: `/v3/users/6190ca8b8acd4e99b228f0851e2ed5af` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:14.579066 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5803d0af-4a85-46e7-a24e-631747da5cbb None admin] RBAC: Authorizing `identity:get_user(user_id=6190ca8b8acd4e99b228f0851e2ed5af)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:14.582556 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5803d0af-4a85-46e7-a24e-631747da5cbb None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:14.585380 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 61/123] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:14 2022] GET /identity/v3/users/6190ca8b8acd4e99b228f0851e2ed5af => generated 247 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:14.588131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ac1bed-956f-44e4-8182-9c4251199fc1 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:14.590239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5803d0af-4a85-46e7-a24e-631747da5cbb None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:14.649072 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9a8a7d-0c23-4531-8aad-7e83abc4a038 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:14.650147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9a8a7d-0c23-4531-8aad-7e83abc4a038 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:14.650663 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9a8a7d-0c23-4531-8aad-7e83abc4a038 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:14.652466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9a8a7d-0c23-4531-8aad-7e83abc4a038 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:14.668953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:14.674080 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9a8a7d-0c23-4531-8aad-7e83abc4a038 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:14.680475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-dd9a8a7d-0c23-4531-8aad-7e83abc4a038 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:06:14.680475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:14.680475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:14.680475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:14.680475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:14.680475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:14.680475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:14.680475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:14.680475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:14.680475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:14.680475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:14.680475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:14.680475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:14.680475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:14.680475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:14.680475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:14.681522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:14.681522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:14.681522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:14.681522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:14.681522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:14.681522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:14.681522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:14.681522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:14.681522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:14.681522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:14.681522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:14.681522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:14.681522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:14.681522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:14.681522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:14.681522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:14.682495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:14.682495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:14.682495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:14.682495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:14.682495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:14.682495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:14.682495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:14.682495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:14.682495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:14.682495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:14.682495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:06:14.682495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:06:14.685617 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 63/124] 10.222.0.26 () {56 vars in 1114 bytes} [Tue Sep 27 04:06:14 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:14.687653 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9a8a7d-0c23-4531-8aad-7e83abc4a038 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:14.775988 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5803d0af-4a85-46e7-a24e-631747da5cbb None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:14.811307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c23932-e563-4ead-a19f-15512d83bd5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:14.811943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c23932-e563-4ead-a19f-15512d83bd5e None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:14.812227 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c23932-e563-4ead-a19f-15512d83bd5e None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:14.812501 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c23932-e563-4ead-a19f-15512d83bd5e None admin] PATH_INFO: `/v3/projects/146fd1189c56413c9d17cd16d1a974d3` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:14.813971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c23932-e563-4ead-a19f-15512d83bd5e None admin] RBAC: Authorizing `identity:get_project(project_id=146fd1189c56413c9d17cd16d1a974d3)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:14.815519 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c23932-e563-4ead-a19f-15512d83bd5e None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:14.817130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 62/125] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:14 2022] GET /identity/v3/projects/146fd1189c56413c9d17cd16d1a974d3 => generated 298 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:14.821955 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9a8a7d-0c23-4531-8aad-7e83abc4a038 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:14.822745 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c23932-e563-4ead-a19f-15512d83bd5e None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:14.858120 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:14.862234 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4f4f3c-9c44-4e81-b613-1965baa50797 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:14.862872 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4f4f3c-9c44-4e81-b613-1965baa50797 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:14.863219 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4f4f3c-9c44-4e81-b613-1965baa50797 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:14.863542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4f4f3c-9c44-4e81-b613-1965baa50797 None admin] PATH_INFO: `/v3/projects/146fd1189c56413c9d17cd16d1a974d3` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:14.865379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4f4f3c-9c44-4e81-b613-1965baa50797 None admin] RBAC: Authorizing `identity:get_project(project_id=146fd1189c56413c9d17cd16d1a974d3)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:14.866878 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4f4f3c-9c44-4e81-b613-1965baa50797 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:14.868757 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 64/126] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:14 2022] GET /identity/v3/projects/146fd1189c56413c9d17cd16d1a974d3 => generated 298 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:14.869539 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4f4f3c-9c44-4e81-b613-1965baa50797 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:14.930440 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c23932-e563-4ead-a19f-15512d83bd5e None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:14.968473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:14.988565 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10f6428-9292-46c0-a183-a8d6bb96ee1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:14.989848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10f6428-9292-46c0-a183-a8d6bb96ee1d None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:14.990281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10f6428-9292-46c0-a183-a8d6bb96ee1d None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:14.990989 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10f6428-9292-46c0-a183-a8d6bb96ee1d None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:14.992401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10f6428-9292-46c0-a183-a8d6bb96ee1d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:14.992973 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10f6428-9292-46c0-a183-a8d6bb96ee1d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:14.994480 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4f4f3c-9c44-4e81-b613-1965baa50797 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:14.994734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10f6428-9292-46c0-a183-a8d6bb96ee1d None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:15.006619 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 63/127] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:06:14 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:15.008164 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10f6428-9292-46c0-a183-a8d6bb96ee1d None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:15.040356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3a9a07-a30a-42ec-a2c0-0a15320cfe35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:15.041497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3a9a07-a30a-42ec-a2c0-0a15320cfe35 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:15.042233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3a9a07-a30a-42ec-a2c0-0a15320cfe35 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:15.042665 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3a9a07-a30a-42ec-a2c0-0a15320cfe35 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:15.057504 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3a9a07-a30a-42ec-a2c0-0a15320cfe35 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:15.064523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-cd3a9a07-a30a-42ec-a2c0-0a15320cfe35 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:15.064523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:15.064523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:15.064523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:15.064523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:15.064523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:15.064523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:15.064523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:15.064523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:15.064523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:15.064523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:15.064523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:15.064523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:15.064523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:15.064523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:15.064523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:15.065589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:15.065589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:15.065589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:15.065589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:15.065589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:15.065589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:15.065589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:15.065589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:15.065589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:15.065589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:15.065589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:15.065589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:15.065589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:15.065589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:15.065589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:15.065589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:15.066682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:15.066682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:15.066682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:15.066682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:15.066682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:15.066682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:15.066682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:15.066682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:15.066682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:15.066682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:15.066682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:15.066682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:06:15.070074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 65/128] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:14 2022] GET /identity/v3/roles/member => generated 84 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:15.071086 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3a9a07-a30a-42ec-a2c0-0a15320cfe35 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:15.120544 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10f6428-9292-46c0-a183-a8d6bb96ee1d None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:15.140583 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:15.158910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e917816-d5b2-40a8-b3fe-accf7af2fb56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:15.159808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e917816-d5b2-40a8-b3fe-accf7af2fb56 None admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:15.160260 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e917816-d5b2-40a8-b3fe-accf7af2fb56 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:15.160684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e917816-d5b2-40a8-b3fe-accf7af2fb56 None admin] PATH_INFO: `/v3/projects/146fd1189c56413c9d17cd16d1a974d3/users/fe29a891245a498fa7af105f29387508/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:15.162888 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e917816-d5b2-40a8-b3fe-accf7af2fb56 None admin] RBAC: Authorizing `identity:create_grant(project_id=146fd1189c56413c9d17cd16d1a974d3, user_id=fe29a891245a498fa7af105f29387508, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:15.164534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e917816-d5b2-40a8-b3fe-accf7af2fb56 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:15.179652 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 64/129] 10.222.0.26 () {58 vars in 1439 bytes} [Tue Sep 27 04:06:15 2022] PUT /identity/v3/projects/146fd1189c56413c9d17cd16d1a974d3/users/fe29a891245a498fa7af105f29387508/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:06:15.181271 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e917816-d5b2-40a8-b3fe-accf7af2fb56 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:15.185481 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3a9a07-a30a-42ec-a2c0-0a15320cfe35 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:15.225406 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ad5313-d74e-48c8-a114-134e2879a163 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:15.226167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ad5313-d74e-48c8-a114-134e2879a163 None admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:15.226458 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ad5313-d74e-48c8-a114-134e2879a163 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:15.226787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ad5313-d74e-48c8-a114-134e2879a163 None admin] PATH_INFO: `/v3/projects/146fd1189c56413c9d17cd16d1a974d3/users/6190ca8b8acd4e99b228f0851e2ed5af/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:15.229067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ad5313-d74e-48c8-a114-134e2879a163 None admin] RBAC: Authorizing `identity:create_grant(project_id=146fd1189c56413c9d17cd16d1a974d3, user_id=6190ca8b8acd4e99b228f0851e2ed5af, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:15.230706 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ad5313-d74e-48c8-a114-134e2879a163 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:15.255496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 66/130] 10.222.0.26 () {58 vars in 1439 bytes} [Tue Sep 27 04:06:15 2022] PUT /identity/v3/projects/146fd1189c56413c9d17cd16d1a974d3/users/6190ca8b8acd4e99b228f0851e2ed5af/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:06:15.256649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ad5313-d74e-48c8-a114-134e2879a163 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:15.258126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:15.401088 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e917816-d5b2-40a8-b3fe-accf7af2fb56 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:15.457712 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ad5313-d74e-48c8-a114-134e2879a163 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:15.459933 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce68d1a6-0d95-4f6b-aff4-80c69d0a345b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:15.460789 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce68d1a6-0d95-4f6b-aff4-80c69d0a345b None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:15.461201 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce68d1a6-0d95-4f6b-aff4-80c69d0a345b None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:15.461625 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce68d1a6-0d95-4f6b-aff4-80c69d0a345b None admin] PATH_INFO: `/v3/users/fe29a891245a498fa7af105f29387508` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:15.463889 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce68d1a6-0d95-4f6b-aff4-80c69d0a345b None admin] RBAC: Authorizing `identity:get_user(user_id=fe29a891245a498fa7af105f29387508)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:15.465518 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce68d1a6-0d95-4f6b-aff4-80c69d0a345b None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:15.467309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 65/131] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:15 2022] GET /identity/v3/users/fe29a891245a498fa7af105f29387508 => generated 275 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:15.468657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8320dc-9d5f-4412-ac7e-38a83685c83c None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:15.469095 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8320dc-9d5f-4412-ac7e-38a83685c83c None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:15.469513 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8320dc-9d5f-4412-ac7e-38a83685c83c None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:15.470953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 66/132] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:15.476538 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8320dc-9d5f-4412-ac7e-38a83685c83c None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:15.494531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:15.508869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d478054-5a60-4871-9834-37d22ad4192f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:15.509496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d478054-5a60-4871-9834-37d22ad4192f None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:15.509813 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d478054-5a60-4871-9834-37d22ad4192f None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:15.510100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d478054-5a60-4871-9834-37d22ad4192f None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:15.510940 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d478054-5a60-4871-9834-37d22ad4192f None admin] RBAC: Adding query filter params (name=member) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:15.511254 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d478054-5a60-4871-9834-37d22ad4192f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:15.512774 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d478054-5a60-4871-9834-37d22ad4192f None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:15.522517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 67/133] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:15 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:15.523742 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50aa2542-ce37-4db5-9526-fc24429cb9e2 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:15.524145 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50aa2542-ce37-4db5-9526-fc24429cb9e2 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:15.524425 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50aa2542-ce37-4db5-9526-fc24429cb9e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:15.549104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-50aa2542-ce37-4db5-9526-fc24429cb9e2 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:15.550337 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-50aa2542-ce37-4db5-9526-fc24429cb9e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:15.603594 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8320dc-9d5f-4412-ac7e-38a83685c83c None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:15.619952 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 68/134] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:15.620811 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50aa2542-ce37-4db5-9526-fc24429cb9e2 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:15.621956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:15.657003 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f4c884-b53d-49ad-8317-0d9689489b49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:15.658024 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f4c884-b53d-49ad-8317-0d9689489b49 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:15.658557 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f4c884-b53d-49ad-8317-0d9689489b49 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:15.659415 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f4c884-b53d-49ad-8317-0d9689489b49 None admin] PATH_INFO: `/v3/projects/146fd1189c56413c9d17cd16d1a974d3` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:15.662837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f4c884-b53d-49ad-8317-0d9689489b49 None admin] RBAC: Authorizing `identity:get_project(project_id=146fd1189c56413c9d17cd16d1a974d3)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:15.665027 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f4c884-b53d-49ad-8317-0d9689489b49 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:15.667006 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 67/135] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:15 2022] GET /identity/v3/projects/146fd1189c56413c9d17cd16d1a974d3 => generated 298 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:15.673130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f4c884-b53d-49ad-8317-0d9689489b49 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:15.839327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50aa2542-ce37-4db5-9526-fc24429cb9e2 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:15.849431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f4c884-b53d-49ad-8317-0d9689489b49 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:15.870211 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:15.906634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183a3ae5-db7b-4c90-88d4-05cae2ebacdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:15.907338 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183a3ae5-db7b-4c90-88d4-05cae2ebacdb None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:15.907569 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183a3ae5-db7b-4c90-88d4-05cae2ebacdb None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:15.907843 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183a3ae5-db7b-4c90-88d4-05cae2ebacdb None admin] PATH_INFO: `/v3/users/fe29a891245a498fa7af105f29387508` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:15.908979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183a3ae5-db7b-4c90-88d4-05cae2ebacdb None admin] RBAC: Authorizing `identity:get_user(user_id=fe29a891245a498fa7af105f29387508)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:15.910460 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183a3ae5-db7b-4c90-88d4-05cae2ebacdb None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:15.912909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 69/136] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:15 2022] GET /identity/v3/users/fe29a891245a498fa7af105f29387508 => generated 275 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:15.913259 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9a42af-f085-4ca9-84f5-07f073e4227f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:15.914369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9a42af-f085-4ca9-84f5-07f073e4227f None admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:15.914655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9a42af-f085-4ca9-84f5-07f073e4227f None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:15.914932 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9a42af-f085-4ca9-84f5-07f073e4227f None admin] PATH_INFO: `/v3/projects/146fd1189c56413c9d17cd16d1a974d3/users/fe29a891245a498fa7af105f29387508/roles/2d9f391485604d0b952b7c844c3a3266` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:15.917086 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9a42af-f085-4ca9-84f5-07f073e4227f None admin] RBAC: Authorizing `identity:create_grant(project_id=146fd1189c56413c9d17cd16d1a974d3, user_id=fe29a891245a498fa7af105f29387508, role_id=2d9f391485604d0b952b7c844c3a3266)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:15.920684 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183a3ae5-db7b-4c90-88d4-05cae2ebacdb None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:15.922318 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9a42af-f085-4ca9-84f5-07f073e4227f None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:15.945941 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 68/137] 10.222.0.26 () {58 vars in 1439 bytes} [Tue Sep 27 04:06:15 2022] PUT /identity/v3/projects/146fd1189c56413c9d17cd16d1a974d3/users/fe29a891245a498fa7af105f29387508/roles/2d9f391485604d0b952b7c844c3a3266 => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:06:16.041891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183a3ae5-db7b-4c90-88d4-05cae2ebacdb None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:16.085623 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c92e93-89ad-4d18-abb6-012095210978 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:16.086309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c92e93-89ad-4d18-abb6-012095210978 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:16.086592 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c92e93-89ad-4d18-abb6-012095210978 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:16.086925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c92e93-89ad-4d18-abb6-012095210978 None admin] PATH_INFO: `/v3/projects/4a259a88e57d4f71bf57358d1fce3cc3` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:16.090438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c92e93-89ad-4d18-abb6-012095210978 None admin] RBAC: Authorizing `identity:get_project(project_id=4a259a88e57d4f71bf57358d1fce3cc3)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:16.091902 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c92e93-89ad-4d18-abb6-012095210978 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:16.093893 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 70/138] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:15 2022] GET /identity/v3/projects/4a259a88e57d4f71bf57358d1fce3cc3 => generated 312 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:16.100291 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9a42af-f085-4ca9-84f5-07f073e4227f None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:16.103819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c92e93-89ad-4d18-abb6-012095210978 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:16.151921 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:16.178212 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:16.218200 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c92e93-89ad-4d18-abb6-012095210978 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:16.283898 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9a42af-f085-4ca9-84f5-07f073e4227f None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:16.294504 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40761284-1ac7-4b15-99b0-af7ab9bb2f0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:16.295174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40761284-1ac7-4b15-99b0-af7ab9bb2f0a None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:16.295545 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40761284-1ac7-4b15-99b0-af7ab9bb2f0a None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:16.295824 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40761284-1ac7-4b15-99b0-af7ab9bb2f0a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:16.305285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40761284-1ac7-4b15-99b0-af7ab9bb2f0a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:16.306839 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40761284-1ac7-4b15-99b0-af7ab9bb2f0a None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:16.308510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 71/139] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:06:16 2022] GET /identity/v3/domains/default => generated 199 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:16.315656 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40761284-1ac7-4b15-99b0-af7ab9bb2f0a None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:16.336644 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41638f03-12a7-4936-8a2d-7c30af52b521 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:16.337430 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41638f03-12a7-4936-8a2d-7c30af52b521 None admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:16.337721 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41638f03-12a7-4936-8a2d-7c30af52b521 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:16.338029 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41638f03-12a7-4936-8a2d-7c30af52b521 None admin] PATH_INFO: `/v3/projects/4a259a88e57d4f71bf57358d1fce3cc3/users/fe29a891245a498fa7af105f29387508/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:16.339959 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41638f03-12a7-4936-8a2d-7c30af52b521 None admin] RBAC: Authorizing `identity:create_grant(project_id=4a259a88e57d4f71bf57358d1fce3cc3, user_id=fe29a891245a498fa7af105f29387508, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:16.341439 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41638f03-12a7-4936-8a2d-7c30af52b521 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:16.361890 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 69/140] 10.222.0.26 () {58 vars in 1439 bytes} [Tue Sep 27 04:06:16 2022] PUT /identity/v3/projects/4a259a88e57d4f71bf57358d1fce3cc3/users/fe29a891245a498fa7af105f29387508/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:06:16.466158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40761284-1ac7-4b15-99b0-af7ab9bb2f0a None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:16.486146 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:16.510505 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15380ad7-53bf-45ee-96c6-cb7a44b299ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:16.511128 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15380ad7-53bf-45ee-96c6-cb7a44b299ac None admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:16.511412 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15380ad7-53bf-45ee-96c6-cb7a44b299ac None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:16.511693 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15380ad7-53bf-45ee-96c6-cb7a44b299ac None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:16.512418 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15380ad7-53bf-45ee-96c6-cb7a44b299ac None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:16.513878 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15380ad7-53bf-45ee-96c6-cb7a44b299ac None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:16.550657 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 72/141] 10.222.0.26 () {60 vars in 1132 bytes} [Tue Sep 27 04:06:16 2022] POST /identity/v3/users => generated 283 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:06:16.698115 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1099b1ec-dc1d-481d-bc86-df4a120020dc None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:16.698631 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1099b1ec-dc1d-481d-bc86-df4a120020dc None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:16.698631 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1099b1ec-dc1d-481d-bc86-df4a120020dc None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:16.700021 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 70/142] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:16.705941 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7847a0-4653-4076-a571-4ed5a36b92a5 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:16.706233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7847a0-4653-4076-a571-4ed5a36b92a5 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:16.706514 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7847a0-4653-4076-a571-4ed5a36b92a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:16.732346 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-4b7847a0-4653-4076-a571-4ed5a36b92a5 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:16.733606 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-4b7847a0-4653-4076-a571-4ed5a36b92a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:16.813102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 73/143] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:17.129244 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1099b1ec-dc1d-481d-bc86-df4a120020dc None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:17.247179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:17.250883 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1099b1ec-dc1d-481d-bc86-df4a120020dc None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:17.293274 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb87246-085f-411d-92b5-c70c0328f70c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:17.293937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb87246-085f-411d-92b5-c70c0328f70c None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:17.294224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb87246-085f-411d-92b5-c70c0328f70c None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:17.294503 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb87246-085f-411d-92b5-c70c0328f70c None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:17.308912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb87246-085f-411d-92b5-c70c0328f70c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:17.313781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-3cb87246-085f-411d-92b5-c70c0328f70c None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 04:06:17.313781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:17.313781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:17.313781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:17.313781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:17.313781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:17.313781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:17.313781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:17.313781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:17.313781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:17.313781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:17.313781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:17.313781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:17.313781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:17.313781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:17.313781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:17.314591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:17.314591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:17.314591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:17.314591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:17.314591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:17.314591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:17.314591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:17.314591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:17.314591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:17.314591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:17.314591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:17.314591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:17.314591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:17.314591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:17.314591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:17.314591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:17.315423 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:17.315423 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:17.315423 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:17.315423 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:17.315423 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:17.315423 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:17.315423 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:17.315423 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:17.315423 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:17.315423 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:17.315423 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 04:06:17.315423 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:06:17.315423 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 71/144] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:17 2022] GET /identity/v3/roles/reader => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:17.321722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7847a0-4653-4076-a571-4ed5a36b92a5 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:17.465655 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:17.469790 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7847a0-4653-4076-a571-4ed5a36b92a5 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:17.509890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0dfd9da-d9af-4828-b0a0-42e91c17c88f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:17.511985 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0dfd9da-d9af-4828-b0a0-42e91c17c88f None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:17.512263 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0dfd9da-d9af-4828-b0a0-42e91c17c88f None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:17.512533 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0dfd9da-d9af-4828-b0a0-42e91c17c88f None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:17.513206 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0dfd9da-d9af-4828-b0a0-42e91c17c88f None admin] RBAC: Adding query filter params (name=reader) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:17.513505 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0dfd9da-d9af-4828-b0a0-42e91c17c88f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:17.514921 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0dfd9da-d9af-4828-b0a0-42e91c17c88f None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:17.524456 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 74/145] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:17 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:17.530054 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb87246-085f-411d-92b5-c70c0328f70c None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:17.655022 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb87246-085f-411d-92b5-c70c0328f70c None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:17.696656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9401992e-e358-4c64-afb3-046adc51ab35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:17.697214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9401992e-e358-4c64-afb3-046adc51ab35 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:17.697488 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9401992e-e358-4c64-afb3-046adc51ab35 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:17.697771 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9401992e-e358-4c64-afb3-046adc51ab35 None admin] PATH_INFO: `/v3/users/71826307b6884fe78abc8b76724601d4` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:17.698924 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9401992e-e358-4c64-afb3-046adc51ab35 None admin] RBAC: Authorizing `identity:get_user(user_id=71826307b6884fe78abc8b76724601d4)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:17.700361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9401992e-e358-4c64-afb3-046adc51ab35 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:17.702434 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 72/146] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:17 2022] GET /identity/v3/users/71826307b6884fe78abc8b76724601d4 => generated 289 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:17.707535 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0dfd9da-d9af-4828-b0a0-42e91c17c88f None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:17.830863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0dfd9da-d9af-4828-b0a0-42e91c17c88f None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:17.866751 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ea6725-d9e7-499d-9134-1c8914e456c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:17.867387 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ea6725-d9e7-499d-9134-1c8914e456c8 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:17.867664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ea6725-d9e7-499d-9134-1c8914e456c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:17.867940 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ea6725-d9e7-499d-9134-1c8914e456c8 None admin] PATH_INFO: `/v3/projects/146fd1189c56413c9d17cd16d1a974d3` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:17.869065 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ea6725-d9e7-499d-9134-1c8914e456c8 None admin] RBAC: Authorizing `identity:get_project(project_id=146fd1189c56413c9d17cd16d1a974d3)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:17.870581 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ea6725-d9e7-499d-9134-1c8914e456c8 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:17.872548 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 75/147] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:17 2022] GET /identity/v3/projects/146fd1189c56413c9d17cd16d1a974d3 => generated 298 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:17.878120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9401992e-e358-4c64-afb3-046adc51ab35 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:17.912051 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:17.997620 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9401992e-e358-4c64-afb3-046adc51ab35 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:18.032913 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f918b3ad-5640-4394-9f64-7d92c18ac027 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:18.033582 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f918b3ad-5640-4394-9f64-7d92c18ac027 None admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:18.033880 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f918b3ad-5640-4394-9f64-7d92c18ac027 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:18.034151 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f918b3ad-5640-4394-9f64-7d92c18ac027 None admin] PATH_INFO: `/v3/projects/146fd1189c56413c9d17cd16d1a974d3/users/71826307b6884fe78abc8b76724601d4/roles/9b729a0f423e47f7bc562c000780ef7b` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:18.036047 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f918b3ad-5640-4394-9f64-7d92c18ac027 None admin] RBAC: Authorizing `identity:create_grant(project_id=146fd1189c56413c9d17cd16d1a974d3, user_id=71826307b6884fe78abc8b76724601d4, role_id=9b729a0f423e47f7bc562c000780ef7b)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:18.038104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f918b3ad-5640-4394-9f64-7d92c18ac027 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:18.055798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 73/148] 10.222.0.26 () {58 vars in 1439 bytes} [Tue Sep 27 04:06:17 2022] PUT /identity/v3/projects/146fd1189c56413c9d17cd16d1a974d3/users/71826307b6884fe78abc8b76724601d4/roles/9b729a0f423e47f7bc562c000780ef7b => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:06:18.563141 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb7e3cb-928f-4227-858c-7d692024c4cd None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:18.563636 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb7e3cb-928f-4227-858c-7d692024c4cd None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:18.563737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb7e3cb-928f-4227-858c-7d692024c4cd None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:18.565273 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 76/149] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:18.575192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a2d9ae-c558-4a5e-81a6-f50ec4e4a590 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:18.575477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a2d9ae-c558-4a5e-81a6-f50ec4e4a590 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:18.575751 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a2d9ae-c558-4a5e-81a6-f50ec4e4a590 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:18.599933 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-69a2d9ae-c558-4a5e-81a6-f50ec4e4a590 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:18.601115 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-69a2d9ae-c558-4a5e-81a6-f50ec4e4a590 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:18.685439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f63a1a0-ad59-42cc-b55d-cb112003df83 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:18.685718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f63a1a0-ad59-42cc-b55d-cb112003df83 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:18.685926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f63a1a0-ad59-42cc-b55d-cb112003df83 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:18.687543 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 77/150] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:18.694459 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2a467c-4c97-40c1-b769-da42359b39ad None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:18.694743 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2a467c-4c97-40c1-b769-da42359b39ad None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:18.695011 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2a467c-4c97-40c1-b769-da42359b39ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:18.697217 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 74/151] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:18.721394 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-da2a467c-4c97-40c1-b769-da42359b39ad None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:18.722608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-da2a467c-4c97-40c1-b769-da42359b39ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:18.747899 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:18.796166 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 78/152] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Sep 27 04:06:19.246674 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a2d9ae-c558-4a5e-81a6-f50ec4e4a590 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:19.275307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:19.376480 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a2d9ae-c558-4a5e-81a6-f50ec4e4a590 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:19.387795 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2a467c-4c97-40c1-b769-da42359b39ad None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:19.448451 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf954877-2c2f-4702-ae8d-6a47417a4f13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:19.449092 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf954877-2c2f-4702-ae8d-6a47417a4f13 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:19.450340 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf954877-2c2f-4702-ae8d-6a47417a4f13 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:19.450661 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf954877-2c2f-4702-ae8d-6a47417a4f13 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:19.474620 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf954877-2c2f-4702-ae8d-6a47417a4f13 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:19.479776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-cf954877-2c2f-4702-ae8d-6a47417a4f13 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:06:19.479776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:19.479776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:19.479776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:19.479776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:19.479776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:19.479776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:19.479776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:19.479776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:19.479776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:19.479776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:19.479776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:19.479776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:19.479776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:19.479776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:19.479776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:19.480596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:19.480596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:19.480596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:19.480596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:19.480596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:19.480596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:19.480596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:19.480596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:19.480596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:19.480596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:19.480596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:19.480596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:19.480596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:19.480596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:19.480596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:19.480596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:19.481344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:19.481344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:19.481344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:19.481344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:19.481344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:19.481344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:19.481344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:19.481344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:19.481344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:19.481344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:19.481344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:06:19.481344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:06:19.483565 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 75/153] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:06:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:19.489074 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf954877-2c2f-4702-ae8d-6a47417a4f13 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:19.529755 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2a467c-4c97-40c1-b769-da42359b39ad None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:19.570593 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d6f00a-35b6-441e-8983-a41ff4c922ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:19.571907 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d6f00a-35b6-441e-8983-a41ff4c922ea None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:19.572320 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d6f00a-35b6-441e-8983-a41ff4c922ea None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:19.572722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d6f00a-35b6-441e-8983-a41ff4c922ea None admin] PATH_INFO: `/v3/roles/member` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:19.585737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d6f00a-35b6-441e-8983-a41ff4c922ea None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:19.591544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-99d6f00a-35b6-441e-8983-a41ff4c922ea None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:19.591544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:19.591544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:19.591544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:19.591544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:19.591544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:19.591544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:19.591544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:19.591544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:19.591544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:19.591544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:19.591544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:19.591544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:19.591544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:19.591544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:19.591544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:19.592415 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:19.592415 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:19.592415 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:19.592415 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:19.592415 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:19.592415 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:19.592415 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:19.592415 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:19.592415 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:19.592415 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:19.592415 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:19.592415 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:19.592415 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:19.592415 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:19.592415 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:19.592415 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:19.593310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:19.593310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:19.593310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:19.593310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:19.593310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:19.593310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:19.593310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:19.593310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:19.593310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:19.593310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:19.593310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:19.593310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:06:19.595933 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 79/154] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:19 2022] GET /identity/v3/roles/member => generated 84 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:19.602201 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d6f00a-35b6-441e-8983-a41ff4c922ea None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:19.623366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:19.664157 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:19.716452 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d6f00a-35b6-441e-8983-a41ff4c922ea None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:19.733526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf954877-2c2f-4702-ae8d-6a47417a4f13 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:19.758366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab2571e-9dec-4524-8092-82456442b021 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:19.759203 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab2571e-9dec-4524-8092-82456442b021 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:19.759625 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab2571e-9dec-4524-8092-82456442b021 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:19.760031 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab2571e-9dec-4524-8092-82456442b021 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:19.760853 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab2571e-9dec-4524-8092-82456442b021 None admin] RBAC: Adding query filter params (name=member) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:19.761289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab2571e-9dec-4524-8092-82456442b021 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:19.763245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab2571e-9dec-4524-8092-82456442b021 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:19.773623 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 80/155] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:19 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:19.775225 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad44adea-9900-4748-8a62-c7e3e47877fc None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:19.775647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad44adea-9900-4748-8a62-c7e3e47877fc None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:19.776078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad44adea-9900-4748-8a62-c7e3e47877fc None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:19.777668 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 81/156] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:19.782964 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31292634-0054-4e5a-bec9-87b44d1ba17f None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:19.783469 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31292634-0054-4e5a-bec9-87b44d1ba17f None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:19.783876 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31292634-0054-4e5a-bec9-87b44d1ba17f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:19.808325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-31292634-0054-4e5a-bec9-87b44d1ba17f None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:19.809720 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-31292634-0054-4e5a-bec9-87b44d1ba17f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:19.847332 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51647cab-71f8-4491-a95f-9e77dcc5105d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:19.847972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51647cab-71f8-4491-a95f-9e77dcc5105d None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:19.848252 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51647cab-71f8-4491-a95f-9e77dcc5105d None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:19.848528 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51647cab-71f8-4491-a95f-9e77dcc5105d None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:19.849268 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51647cab-71f8-4491-a95f-9e77dcc5105d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:19.849574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51647cab-71f8-4491-a95f-9e77dcc5105d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:19.853777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51647cab-71f8-4491-a95f-9e77dcc5105d None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:19.874173 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 76/157] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 383 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:19.874173 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51647cab-71f8-4491-a95f-9e77dcc5105d None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:19.916554 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 82/158] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:19.917703 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31292634-0054-4e5a-bec9-87b44d1ba17f None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:19.930521 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:20.048940 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51647cab-71f8-4491-a95f-9e77dcc5105d None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:20.069257 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31292634-0054-4e5a-bec9-87b44d1ba17f None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:20.101273 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa58bed-c958-4148-b745-4514b01371b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:20.101887 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa58bed-c958-4148-b745-4514b01371b6 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:20.102171 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa58bed-c958-4148-b745-4514b01371b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:20.102449 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa58bed-c958-4148-b745-4514b01371b6 None admin] PATH_INFO: `/v3/users/fe29a891245a498fa7af105f29387508` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:20.103593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa58bed-c958-4148-b745-4514b01371b6 None admin] RBAC: Authorizing `identity:get_user(user_id=fe29a891245a498fa7af105f29387508)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:20.105050 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa58bed-c958-4148-b745-4514b01371b6 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:20.106700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 77/159] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:19 2022] GET /identity/v3/users/fe29a891245a498fa7af105f29387508 => generated 275 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:20.112292 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa58bed-c958-4148-b745-4514b01371b6 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:20.121156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ded7142-16f0-4421-a61d-cb7efd16cf0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:20.122017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ded7142-16f0-4421-a61d-cb7efd16cf0d None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:20.122432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ded7142-16f0-4421-a61d-cb7efd16cf0d None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:20.122840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ded7142-16f0-4421-a61d-cb7efd16cf0d None admin] PATH_INFO: `/v3/users/6190ca8b8acd4e99b228f0851e2ed5af` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:20.124131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ded7142-16f0-4421-a61d-cb7efd16cf0d None admin] RBAC: Authorizing `identity:get_user(user_id=6190ca8b8acd4e99b228f0851e2ed5af)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:20.125692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ded7142-16f0-4421-a61d-cb7efd16cf0d None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:20.127796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 83/160] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:19 2022] GET /identity/v3/users/6190ca8b8acd4e99b228f0851e2ed5af => generated 247 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:20.133495 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:20.134730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ded7142-16f0-4421-a61d-cb7efd16cf0d None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:20.218845 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:20.243999 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ded7142-16f0-4421-a61d-cb7efd16cf0d None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:20.254129 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa58bed-c958-4148-b745-4514b01371b6 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:20.282283 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cb0bb7-c671-430f-b190-29086d86cfdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:20.283149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cb0bb7-c671-430f-b190-29086d86cfdc None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:20.283569 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cb0bb7-c671-430f-b190-29086d86cfdc None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:20.283978 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cb0bb7-c671-430f-b190-29086d86cfdc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:20.311490 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83853a13-044a-4998-a2a3-e99dd2e411a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:20.312031 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83853a13-044a-4998-a2a3-e99dd2e411a7 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:20.312310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83853a13-044a-4998-a2a3-e99dd2e411a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:20.312582 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83853a13-044a-4998-a2a3-e99dd2e411a7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:20.357779 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53cb0bb7-c671-430f-b190-29086d86cfdc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:20.359478 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53cb0bb7-c671-430f-b190-29086d86cfdc None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:20.412396 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83853a13-044a-4998-a2a3-e99dd2e411a7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:20.413920 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83853a13-044a-4998-a2a3-e99dd2e411a7 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:20.443532 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:20.446104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:20.468124 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 84/161] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 27 04:06:20.476816 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cb0bb7-c671-430f-b190-29086d86cfdc None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:20.617253 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 78/162] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 505 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:06:20.618534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83853a13-044a-4998-a2a3-e99dd2e411a7 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:20.698679 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cb0bb7-c671-430f-b190-29086d86cfdc None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:20.782888 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d95018-0ce8-44fa-966c-b057e084cf12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:20.783514 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d95018-0ce8-44fa-966c-b057e084cf12 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:20.783787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d95018-0ce8-44fa-966c-b057e084cf12 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:20.784040 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d95018-0ce8-44fa-966c-b057e084cf12 None admin] PATH_INFO: `/v3/projects/146fd1189c56413c9d17cd16d1a974d3` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:20.785479 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d95018-0ce8-44fa-966c-b057e084cf12 None admin] RBAC: Authorizing `identity:get_project(project_id=146fd1189c56413c9d17cd16d1a974d3)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:20.787037 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d95018-0ce8-44fa-966c-b057e084cf12 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:20.788660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 85/163] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:20 2022] GET /identity/v3/projects/146fd1189c56413c9d17cd16d1a974d3 => generated 298 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:20.789413 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d95018-0ce8-44fa-966c-b057e084cf12 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:20.815248 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83853a13-044a-4998-a2a3-e99dd2e411a7 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:20.830133 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:20.869897 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:20.875666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b3f0a5-6b57-4d45-87d7-357627a0a507 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:20.880802 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b3f0a5-6b57-4d45-87d7-357627a0a507 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:20.880802 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b3f0a5-6b57-4d45-87d7-357627a0a507 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:20.880802 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b3f0a5-6b57-4d45-87d7-357627a0a507 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:20.893396 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b3f0a5-6b57-4d45-87d7-357627a0a507 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:20.899764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-e9b3f0a5-6b57-4d45-87d7-357627a0a507 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:06:20.899764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:20.899764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:20.899764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:20.899764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:20.899764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:20.899764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:20.899764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:20.899764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:20.899764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:20.899764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:20.899764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:20.899764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:20.899764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:20.899764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:20.899764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:20.900703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:20.900703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:20.900703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:20.900703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:20.900703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:20.900703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:20.900703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:20.900703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:20.900703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:20.900703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:20.900703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:20.900703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:20.900703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:20.900703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:20.900703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:20.900703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:20.901492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:20.901492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:20.901492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:20.901492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:20.901492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:20.901492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:20.901492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:20.901492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:20.901492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:20.901492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:20.901492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:06:20.901492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:06:20.904189 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 79/164] 10.222.0.26 () {56 vars in 1114 bytes} [Tue Sep 27 04:06:20 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:20.905492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b3f0a5-6b57-4d45-87d7-357627a0a507 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:21.000557 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d95018-0ce8-44fa-966c-b057e084cf12 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:21.037870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d484f6-e4ef-4465-8f1a-44bbbb1c4f3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:21.038862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d484f6-e4ef-4465-8f1a-44bbbb1c4f3c None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:21.039348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d484f6-e4ef-4465-8f1a-44bbbb1c4f3c None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:21.039829 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d484f6-e4ef-4465-8f1a-44bbbb1c4f3c None admin] PATH_INFO: `/v3/projects/146fd1189c56413c9d17cd16d1a974d3` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:21.041142 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d484f6-e4ef-4465-8f1a-44bbbb1c4f3c None admin] RBAC: Authorizing `identity:get_project(project_id=146fd1189c56413c9d17cd16d1a974d3)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:21.043034 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d484f6-e4ef-4465-8f1a-44bbbb1c4f3c None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:21.044905 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 86/165] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:20 2022] GET /identity/v3/projects/146fd1189c56413c9d17cd16d1a974d3 => generated 298 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:21.046138 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d484f6-e4ef-4465-8f1a-44bbbb1c4f3c None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:21.058030 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b3f0a5-6b57-4d45-87d7-357627a0a507 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:21.093384 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c9dacb-37de-4c98-a6ad-8fd9b20c9ba0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:21.094040 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c9dacb-37de-4c98-a6ad-8fd9b20c9ba0 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:21.094315 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c9dacb-37de-4c98-a6ad-8fd9b20c9ba0 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:21.094562 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c9dacb-37de-4c98-a6ad-8fd9b20c9ba0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:21.095283 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8c9dacb-37de-4c98-a6ad-8fd9b20c9ba0 None admin] RBAC: Adding query filter params (role.id=e6a31562a18649f0809202c36a8e18ef, scope.project.id=146fd1189c56413c9d17cd16d1a974d3, user.id=6190ca8b8acd4e99b228f0851e2ed5af) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:21.095584 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8c9dacb-37de-4c98-a6ad-8fd9b20c9ba0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:21.097157 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8c9dacb-37de-4c98-a6ad-8fd9b20c9ba0 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:21.103778 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 80/166] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:06:20 2022] GET /identity/v3/role_assignments?user.id=6190ca8b8acd4e99b228f0851e2ed5af&scope.project.id=146fd1189c56413c9d17cd16d1a974d3&role.id=e6a31562a18649f0809202c36a8e18ef => generated 608 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:21.104634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c9dacb-37de-4c98-a6ad-8fd9b20c9ba0 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:21.170551 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:21.183684 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d484f6-e4ef-4465-8f1a-44bbbb1c4f3c None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:21.200360 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:21.224519 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f42d0f-1346-4eb7-9b07-a2cbe33c4e17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:21.225496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f42d0f-1346-4eb7-9b07-a2cbe33c4e17 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:21.226011 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f42d0f-1346-4eb7-9b07-a2cbe33c4e17 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:21.226505 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f42d0f-1346-4eb7-9b07-a2cbe33c4e17 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:21.227378 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f42d0f-1346-4eb7-9b07-a2cbe33c4e17 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:21.227897 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f42d0f-1346-4eb7-9b07-a2cbe33c4e17 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:21.229512 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f42d0f-1346-4eb7-9b07-a2cbe33c4e17 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:21.238997 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 87/167] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:06:21 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:21.246914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f42d0f-1346-4eb7-9b07-a2cbe33c4e17 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:21.255329 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c9dacb-37de-4c98-a6ad-8fd9b20c9ba0 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:21.292754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfd3680-e89c-4928-bceb-a1e648b4d20c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:21.293397 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfd3680-e89c-4928-bceb-a1e648b4d20c None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:21.293652 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfd3680-e89c-4928-bceb-a1e648b4d20c None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:21.293975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfd3680-e89c-4928-bceb-a1e648b4d20c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:21.294873 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfd3680-e89c-4928-bceb-a1e648b4d20c None admin] RBAC: Adding query filter params (role.id=e3ce5a3697ec4292a446366d3caadeea, scope.project.id=146fd1189c56413c9d17cd16d1a974d3, user.id=fe29a891245a498fa7af105f29387508) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:21.295061 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfd3680-e89c-4928-bceb-a1e648b4d20c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:21.296489 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfd3680-e89c-4928-bceb-a1e648b4d20c None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:21.307020 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 81/168] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:06:21 2022] GET /identity/v3/role_assignments?user.id=fe29a891245a498fa7af105f29387508&scope.project.id=146fd1189c56413c9d17cd16d1a974d3&role.id=e3ce5a3697ec4292a446366d3caadeea => generated 608 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:21.307020 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dde62c2-7f80-4c35-9c1b-37e1490e4cac None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:21.307020 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dde62c2-7f80-4c35-9c1b-37e1490e4cac None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:21.307020 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dde62c2-7f80-4c35-9c1b-37e1490e4cac None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:21.307020 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 82/169] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:21.319254 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7761015a-630f-4b2a-b350-16b18ef4a419 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:21.319867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7761015a-630f-4b2a-b350-16b18ef4a419 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:21.319867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7761015a-630f-4b2a-b350-16b18ef4a419 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:21.346553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-7761015a-630f-4b2a-b350-16b18ef4a419 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:21.347807 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-7761015a-630f-4b2a-b350-16b18ef4a419 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:21.410000 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f42d0f-1346-4eb7-9b07-a2cbe33c4e17 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:21.465486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a5ae05-6502-468d-baa1-a22d45773609 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:21.466075 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a5ae05-6502-468d-baa1-a22d45773609 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:21.466355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a5ae05-6502-468d-baa1-a22d45773609 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:21.466628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a5ae05-6502-468d-baa1-a22d45773609 None admin] PATH_INFO: `/v3/users/fe29a891245a498fa7af105f29387508` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:21.467810 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a5ae05-6502-468d-baa1-a22d45773609 None admin] RBAC: Authorizing `identity:get_user(user_id=fe29a891245a498fa7af105f29387508)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:21.469304 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a5ae05-6502-468d-baa1-a22d45773609 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:21.474484 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 88/170] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:21 2022] GET /identity/v3/users/fe29a891245a498fa7af105f29387508 => generated 275 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:21.477954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a5ae05-6502-468d-baa1-a22d45773609 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:21.551814 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:21.579108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 83/171] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:21.649863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a5ae05-6502-468d-baa1-a22d45773609 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:21.702726 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89051f01-2bb9-4ea4-8ef6-3ddd8ce2c3f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:21.703344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89051f01-2bb9-4ea4-8ef6-3ddd8ce2c3f6 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:21.703607 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89051f01-2bb9-4ea4-8ef6-3ddd8ce2c3f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:21.703885 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89051f01-2bb9-4ea4-8ef6-3ddd8ce2c3f6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:21.754557 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ac49cb-9f90-4584-9dc5-3593bdb267c4 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:21.755427 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ac49cb-9f90-4584-9dc5-3593bdb267c4 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:21.755662 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ac49cb-9f90-4584-9dc5-3593bdb267c4 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:21.757013 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 84/172] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:06:21.775197 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4963effa-d3cb-4688-b7f7-ebca609c910e None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:21.775497 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4963effa-d3cb-4688-b7f7-ebca609c910e None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:21.775746 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4963effa-d3cb-4688-b7f7-ebca609c910e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:21.789083 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89051f01-2bb9-4ea4-8ef6-3ddd8ce2c3f6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:21.790707 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89051f01-2bb9-4ea4-8ef6-3ddd8ce2c3f6 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:21.809785 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-4963effa-d3cb-4688-b7f7-ebca609c910e None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:21.810995 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-4963effa-d3cb-4688-b7f7-ebca609c910e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:21.858289 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:21.870054 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:21.975858 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 85/173] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:21.976587 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4963effa-d3cb-4688-b7f7-ebca609c910e None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:22.009126 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 89/174] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 531 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 27 04:06:22.015261 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89051f01-2bb9-4ea4-8ef6-3ddd8ce2c3f6 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:22.188565 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89051f01-2bb9-4ea4-8ef6-3ddd8ce2c3f6 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:22.193330 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4963effa-d3cb-4688-b7f7-ebca609c910e None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:22.246521 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea19e27-161d-44e0-a461-9ae92a6787e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:22.247559 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea19e27-161d-44e0-a461-9ae92a6787e8 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:22.248025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea19e27-161d-44e0-a461-9ae92a6787e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:22.248498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea19e27-161d-44e0-a461-9ae92a6787e8 None admin] PATH_INFO: `/v3/projects/146fd1189c56413c9d17cd16d1a974d3` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:22.250978 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea19e27-161d-44e0-a461-9ae92a6787e8 None admin] RBAC: Authorizing `identity:get_project(project_id=146fd1189c56413c9d17cd16d1a974d3)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:22.252946 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea19e27-161d-44e0-a461-9ae92a6787e8 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:22.255656 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 90/175] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:22 2022] GET /identity/v3/projects/146fd1189c56413c9d17cd16d1a974d3 => generated 298 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:22.261121 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7861029a-22f0-47fc-ac76-171a74f3c57a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:22.261723 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7861029a-22f0-47fc-ac76-171a74f3c57a None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:22.262035 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7861029a-22f0-47fc-ac76-171a74f3c57a None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:22.262314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7861029a-22f0-47fc-ac76-171a74f3c57a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:22.262973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea19e27-161d-44e0-a461-9ae92a6787e8 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:22.285205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:22.289143 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7861029a-22f0-47fc-ac76-171a74f3c57a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:22.300725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-7861029a-22f0-47fc-ac76-171a74f3c57a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:22.300725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:22.300725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:22.300725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:22.300725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:22.300725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:22.300725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:22.300725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:22.300725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:22.300725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:22.300725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:22.300725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:22.300725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:22.300725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:22.300725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:22.300725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:22.301669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:22.301669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:22.301669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:22.301669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:22.301669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:22.301669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:22.301669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:22.301669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:22.301669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:22.301669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:22.301669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:22.301669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:22.301669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:22.301669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:22.301669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:22.301669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:22.302992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:22.302992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:22.302992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:22.302992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:22.302992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:22.302992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:22.302992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:22.302992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:22.302992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:22.302992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:22.302992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:22.302992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:06:22.302992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 86/176] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:21 2022] GET /identity/v3/roles/member => generated 84 bytes in 326 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:22.303655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf48577e-55dc-4bca-b4d1-63a023a7f6da None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:22.304239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf48577e-55dc-4bca-b4d1-63a023a7f6da None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:22.304239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf48577e-55dc-4bca-b4d1-63a023a7f6da None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:22.306177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 87/177] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:22.313496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf48577e-55dc-4bca-b4d1-63a023a7f6da None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:22.335698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:22.395086 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea19e27-161d-44e0-a461-9ae92a6787e8 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:22.421984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf48577e-55dc-4bca-b4d1-63a023a7f6da None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:22.444246 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283d0a6f-269b-49c6-ad08-5112412dd448 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:22.445225 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283d0a6f-269b-49c6-ad08-5112412dd448 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:22.445712 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283d0a6f-269b-49c6-ad08-5112412dd448 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:22.446296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283d0a6f-269b-49c6-ad08-5112412dd448 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:22.447389 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283d0a6f-269b-49c6-ad08-5112412dd448 None admin] RBAC: Adding query filter params (role.id=2d9f391485604d0b952b7c844c3a3266, scope.project.id=146fd1189c56413c9d17cd16d1a974d3, user.id=fe29a891245a498fa7af105f29387508) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:22.447946 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283d0a6f-269b-49c6-ad08-5112412dd448 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:22.449782 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283d0a6f-269b-49c6-ad08-5112412dd448 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:22.461925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 91/178] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:06:22 2022] GET /identity/v3/role_assignments?user.id=fe29a891245a498fa7af105f29387508&scope.project.id=146fd1189c56413c9d17cd16d1a974d3&role.id=2d9f391485604d0b952b7c844c3a3266 => generated 608 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:22.463338 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105de34a-11c1-466d-a636-e0b369c7f364 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:22.463627 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105de34a-11c1-466d-a636-e0b369c7f364 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:22.463905 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105de34a-11c1-466d-a636-e0b369c7f364 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:22.468866 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e62bc56-c204-4ece-8fdb-5f4d0d8020c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:22.469498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e62bc56-c204-4ece-8fdb-5f4d0d8020c4 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:22.469863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e62bc56-c204-4ece-8fdb-5f4d0d8020c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:22.470142 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e62bc56-c204-4ece-8fdb-5f4d0d8020c4 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:22.470831 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e62bc56-c204-4ece-8fdb-5f4d0d8020c4 None admin] RBAC: Adding query filter params (name=member) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:22.471135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e62bc56-c204-4ece-8fdb-5f4d0d8020c4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:22.472554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e62bc56-c204-4ece-8fdb-5f4d0d8020c4 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:22.493862 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 88/179] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:22 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:22.495043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e62bc56-c204-4ece-8fdb-5f4d0d8020c4 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:22.497308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-105de34a-11c1-466d-a636-e0b369c7f364 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:22.498789 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-105de34a-11c1-466d-a636-e0b369c7f364 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:22.613704 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 92/180] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:22.615590 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105de34a-11c1-466d-a636-e0b369c7f364 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:22.624723 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:22.687878 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:22.716967 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e62bc56-c204-4ece-8fdb-5f4d0d8020c4 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:22.740881 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105de34a-11c1-466d-a636-e0b369c7f364 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:22.761385 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88eb69b4-8a15-4438-bccd-5edb10cbab37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:22.762256 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88eb69b4-8a15-4438-bccd-5edb10cbab37 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:22.762535 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88eb69b4-8a15-4438-bccd-5edb10cbab37 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:22.762802 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88eb69b4-8a15-4438-bccd-5edb10cbab37 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:22.776204 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88eb69b4-8a15-4438-bccd-5edb10cbab37 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:22.783007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-88eb69b4-8a15-4438-bccd-5edb10cbab37 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:06:22.783007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:22.783007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:22.783007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:22.783007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:22.783007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:22.783007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:22.783007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:22.783007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:22.783007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:22.783007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:22.783007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:22.783007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:22.783007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:22.783007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:22.783007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:22.783875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:22.783875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:22.783875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:22.783875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:22.783875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:22.783875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:22.783875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:22.783875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:22.783875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:22.783875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:22.783875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:22.783875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:22.783875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:22.783875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:22.783875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:22.783875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:22.784658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:22.784658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:22.784658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:22.784658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:22.784658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:22.784658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:22.784658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:22.784658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:22.784658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:22.784658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:22.784658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:06:22.784658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:06:22.785306 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939be940-9db5-4c5a-abc8-0d103484667e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:22.785392 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 89/181] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:06:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:22.785464 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939be940-9db5-4c5a-abc8-0d103484667e None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:22.785464 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939be940-9db5-4c5a-abc8-0d103484667e None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:22.785464 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939be940-9db5-4c5a-abc8-0d103484667e None admin] PATH_INFO: `/v3/users/fe29a891245a498fa7af105f29387508` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:22.785650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6efb745-8121-4296-a862-3c5e8dc066c0 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:22.785650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6efb745-8121-4296-a862-3c5e8dc066c0 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:22.786897 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6efb745-8121-4296-a862-3c5e8dc066c0 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:22.787006 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939be940-9db5-4c5a-abc8-0d103484667e None admin] RBAC: Authorizing `identity:get_user(user_id=fe29a891245a498fa7af105f29387508)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:22.787855 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 90/182] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:22.788627 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939be940-9db5-4c5a-abc8-0d103484667e None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:22.788728 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a227119-6190-4dc8-97a2-b113e0c249e3 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:22.789037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a227119-6190-4dc8-97a2-b113e0c249e3 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:22.789345 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a227119-6190-4dc8-97a2-b113e0c249e3 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:22.793871 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 93/183] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:22 2022] GET /identity/v3/users/fe29a891245a498fa7af105f29387508 => generated 275 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:22.794014 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 91/184] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:22.795660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939be940-9db5-4c5a-abc8-0d103484667e None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:22.796515 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3129d06-494c-4adb-a45a-e7a6fc434079 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:22.796809 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3129d06-494c-4adb-a45a-e7a6fc434079 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:22.797084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3129d06-494c-4adb-a45a-e7a6fc434079 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:22.814702 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-a3129d06-494c-4adb-a45a-e7a6fc434079 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:22.815869 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a3129d06-494c-4adb-a45a-e7a6fc434079 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:22.880734 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:22.888266 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 92/185] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:06:22.889394 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896337bf-7b98-43c8-81aa-480bbd245e2b None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:22.889681 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896337bf-7b98-43c8-81aa-480bbd245e2b None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:22.889982 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896337bf-7b98-43c8-81aa-480bbd245e2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:22.912081 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-896337bf-7b98-43c8-81aa-480bbd245e2b None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:22.913050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939be940-9db5-4c5a-abc8-0d103484667e None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:22.913296 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-896337bf-7b98-43c8-81aa-480bbd245e2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:22.949799 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdff5adc-b3c5-4618-8823-43b47383645d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:22.950402 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdff5adc-b3c5-4618-8823-43b47383645d None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:22.950677 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdff5adc-b3c5-4618-8823-43b47383645d None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:22.950942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdff5adc-b3c5-4618-8823-43b47383645d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:22.978986 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:22.990930 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 93/186] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:22.990930 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896337bf-7b98-43c8-81aa-480bbd245e2b None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:23.034033 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdff5adc-b3c5-4618-8823-43b47383645d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:23.035651 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdff5adc-b3c5-4618-8823-43b47383645d None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:23.106257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896337bf-7b98-43c8-81aa-480bbd245e2b None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:23.118681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:23.147027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 94/187] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 27 04:06:23.147792 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdff5adc-b3c5-4618-8823-43b47383645d None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:23.151933 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2b02af-3b83-4504-ad28-b557a1b94407 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:23.152642 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2b02af-3b83-4504-ad28-b557a1b94407 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:23.152945 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2b02af-3b83-4504-ad28-b557a1b94407 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:23.153274 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2b02af-3b83-4504-ad28-b557a1b94407 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:23.154038 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2b02af-3b83-4504-ad28-b557a1b94407 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:23.154397 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2b02af-3b83-4504-ad28-b557a1b94407 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:23.156353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2b02af-3b83-4504-ad28-b557a1b94407 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:23.166566 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 94/188] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:22 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:23.167371 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2b02af-3b83-4504-ad28-b557a1b94407 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:23.224972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:23.253621 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdff5adc-b3c5-4618-8823-43b47383645d None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:23.265942 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2b02af-3b83-4504-ad28-b557a1b94407 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:23.291333 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa898f4-798f-4b9b-b937-99e66739c666 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:23.291964 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa898f4-798f-4b9b-b937-99e66739c666 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:23.292245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa898f4-798f-4b9b-b937-99e66739c666 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:23.292521 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa898f4-798f-4b9b-b937-99e66739c666 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:23.301007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51075128-fb8d-4d9e-a05a-abbc500db535 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:23.301630 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51075128-fb8d-4d9e-a05a-abbc500db535 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:23.301962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51075128-fb8d-4d9e-a05a-abbc500db535 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:23.302232 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51075128-fb8d-4d9e-a05a-abbc500db535 None admin] PATH_INFO: `/v3/projects/4a259a88e57d4f71bf57358d1fce3cc3` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:23.303226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa898f4-798f-4b9b-b937-99e66739c666 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:23.303340 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51075128-fb8d-4d9e-a05a-abbc500db535 None admin] RBAC: Authorizing `identity:get_project(project_id=4a259a88e57d4f71bf57358d1fce3cc3)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:23.304760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51075128-fb8d-4d9e-a05a-abbc500db535 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:23.306390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 95/189] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:23 2022] GET /identity/v3/projects/4a259a88e57d4f71bf57358d1fce3cc3 => generated 312 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:23.307187 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51075128-fb8d-4d9e-a05a-abbc500db535 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:23.308080 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-baa898f4-798f-4b9b-b937-99e66739c666 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:06:23.308080 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:23.308080 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:23.308080 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:23.308080 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:23.308080 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:23.308080 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:23.308080 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:23.308080 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:23.308080 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:23.308080 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:23.308080 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:23.308080 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:23.308080 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:23.308080 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:23.308080 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:23.308956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:23.308956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:23.308956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:23.308956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:23.308956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:23.308956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:23.308956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:23.308956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:23.308956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:23.308956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:23.308956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:23.308956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:23.308956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:23.308956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:23.308956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:23.308956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:23.310282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:23.310282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:23.310282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:23.310282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:23.310282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:23.310282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:23.310282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:23.310282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:23.310282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:23.310282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:23.310282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:06:23.310282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:06:23.310282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 95/190] 10.222.0.26 () {56 vars in 1114 bytes} [Tue Sep 27 04:06:23 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:23.310282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa898f4-798f-4b9b-b937-99e66739c666 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:23.343865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:23.407066 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51075128-fb8d-4d9e-a05a-abbc500db535 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:23.411973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa898f4-798f-4b9b-b937-99e66739c666 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:23.432910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:23.440859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf59b70-7cff-4c63-8f7a-2d9f4d4eabfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:23.441483 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf59b70-7cff-4c63-8f7a-2d9f4d4eabfa None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:23.441778 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf59b70-7cff-4c63-8f7a-2d9f4d4eabfa None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:23.442055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf59b70-7cff-4c63-8f7a-2d9f4d4eabfa None admin] PATH_INFO: `/v3/domains/default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:23.443665 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf59b70-7cff-4c63-8f7a-2d9f4d4eabfa None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:23.445093 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf59b70-7cff-4c63-8f7a-2d9f4d4eabfa None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:23.445184 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ea37a8-f643-495e-8c1a-86c8c4f602d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:23.445821 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ea37a8-f643-495e-8c1a-86c8c4f602d3 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:23.446096 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ea37a8-f643-495e-8c1a-86c8c4f602d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:23.446361 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ea37a8-f643-495e-8c1a-86c8c4f602d3 None admin] PATH_INFO: `/v3/users/1329a7ba9b3d48149f5a05ddeaf4bd31` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:23.446675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 96/191] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:06:23 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:23.447424 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf59b70-7cff-4c63-8f7a-2d9f4d4eabfa None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:23.463292 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ea37a8-f643-495e-8c1a-86c8c4f602d3 None admin] RBAC: Authorizing `identity:get_user(user_id=1329a7ba9b3d48149f5a05ddeaf4bd31)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:23.464735 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ea37a8-f643-495e-8c1a-86c8c4f602d3 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:23.466399 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 96/192] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:23 2022] GET /identity/v3/users/1329a7ba9b3d48149f5a05ddeaf4bd31 => generated 283 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:23.467151 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ea37a8-f643-495e-8c1a-86c8c4f602d3 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:23.547762 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf59b70-7cff-4c63-8f7a-2d9f4d4eabfa None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:23.558816 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:23.568767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ea37a8-f643-495e-8c1a-86c8c4f602d3 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:23.580839 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404fb8d4-8725-4967-90e8-5ced69e15db0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:23.581460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404fb8d4-8725-4967-90e8-5ced69e15db0 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:23.581732 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404fb8d4-8725-4967-90e8-5ced69e15db0 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:23.582032 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404fb8d4-8725-4967-90e8-5ced69e15db0 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:23.591829 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404fb8d4-8725-4967-90e8-5ced69e15db0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:23.597059 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-404fb8d4-8725-4967-90e8-5ced69e15db0 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 04:06:23.597059 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:23.597059 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:23.597059 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:23.597059 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:23.597059 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:23.597059 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:23.597059 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:23.597059 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:23.597059 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:23.597059 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:23.597059 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:23.597059 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:23.597059 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:23.597059 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:23.597059 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:23.597951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:23.597951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:23.597951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:23.597951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:23.597951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:23.597951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:23.597951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:23.597951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:23.597951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:23.597951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:23.597951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:23.597951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:23.597951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:23.597951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:23.597951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:23.597951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:23.598758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:23.598758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:23.598758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:23.598758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:23.598758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:23.598758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:23.598758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:23.598758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:23.598758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:23.598758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:23.598758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 04:06:23.598758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:06:23.598758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 97/193] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:23 2022] GET /identity/v3/roles/reader => generated 84 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:23.599402 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404fb8d4-8725-4967-90e8-5ced69e15db0 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:23.602515 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d08f11f-048d-4143-92b8-4c1a7d80da5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:23.603114 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d08f11f-048d-4143-92b8-4c1a7d80da5f None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:23.603386 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d08f11f-048d-4143-92b8-4c1a7d80da5f None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:23.603654 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d08f11f-048d-4143-92b8-4c1a7d80da5f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:23.604377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d08f11f-048d-4143-92b8-4c1a7d80da5f None admin] RBAC: Adding query filter params (role.id=e3ce5a3697ec4292a446366d3caadeea, scope.project.id=4a259a88e57d4f71bf57358d1fce3cc3, user.id=fe29a891245a498fa7af105f29387508) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:23.604682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d08f11f-048d-4143-92b8-4c1a7d80da5f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:23.606147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d08f11f-048d-4143-92b8-4c1a7d80da5f None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:23.612088 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 97/194] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:06:23 2022] GET /identity/v3/role_assignments?user.id=fe29a891245a498fa7af105f29387508&scope.project.id=4a259a88e57d4f71bf57358d1fce3cc3&role.id=e3ce5a3697ec4292a446366d3caadeea => generated 608 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:23.615096 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d08f11f-048d-4143-92b8-4c1a7d80da5f None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:23.666397 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:23.727277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404fb8d4-8725-4967-90e8-5ced69e15db0 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:23.733683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d08f11f-048d-4143-92b8-4c1a7d80da5f None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:23.767107 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb53dda-057e-4fd0-8b66-47fb07b197a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:23.767842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb53dda-057e-4fd0-8b66-47fb07b197a9 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:23.768189 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb53dda-057e-4fd0-8b66-47fb07b197a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:23.768532 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb53dda-057e-4fd0-8b66-47fb07b197a9 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:23.769284 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb53dda-057e-4fd0-8b66-47fb07b197a9 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:23.769654 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb53dda-057e-4fd0-8b66-47fb07b197a9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:23.770970 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9099a85a-9f64-45ef-bbd6-42ce59d1ccca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:23.771433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb53dda-057e-4fd0-8b66-47fb07b197a9 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:23.771718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9099a85a-9f64-45ef-bbd6-42ce59d1ccca None admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:23.772070 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9099a85a-9f64-45ef-bbd6-42ce59d1ccca None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:23.772412 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9099a85a-9f64-45ef-bbd6-42ce59d1ccca None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:23.773379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9099a85a-9f64-45ef-bbd6-42ce59d1ccca None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:23.775384 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9099a85a-9f64-45ef-bbd6-42ce59d1ccca None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:23.780754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 98/195] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:06:23 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:23.782494 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb53dda-057e-4fd0-8b66-47fb07b197a9 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:23.797927 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 98/196] 10.222.0.26 () {60 vars in 1132 bytes} [Tue Sep 27 04:06:23 2022] POST /identity/v3/users => generated 297 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:06:23.799011 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9099a85a-9f64-45ef-bbd6-42ce59d1ccca None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:23.820549 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:23.932745 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb53dda-057e-4fd0-8b66-47fb07b197a9 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:23.939932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9099a85a-9f64-45ef-bbd6-42ce59d1ccca None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:23.955128 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:23.974601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7220b24-489a-47b9-83a9-a62ae051810a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:23.975220 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7220b24-489a-47b9-83a9-a62ae051810a None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:23.975497 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7220b24-489a-47b9-83a9-a62ae051810a None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:23.975769 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7220b24-489a-47b9-83a9-a62ae051810a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:23.977825 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd276746-6382-4733-95a8-9037d8695da3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:23.978887 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd276746-6382-4733-95a8-9037d8695da3 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:23.978887 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd276746-6382-4733-95a8-9037d8695da3 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:23.979044 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd276746-6382-4733-95a8-9037d8695da3 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:23.979618 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd276746-6382-4733-95a8-9037d8695da3 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:23.979916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd276746-6382-4733-95a8-9037d8695da3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:23.981311 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd276746-6382-4733-95a8-9037d8695da3 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:23.995540 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 99/197] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:23 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:23.995540 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd276746-6382-4733-95a8-9037d8695da3 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:24.063381 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7220b24-489a-47b9-83a9-a62ae051810a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:24.066146 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7220b24-489a-47b9-83a9-a62ae051810a None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:24.088361 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:24.121758 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd276746-6382-4733-95a8-9037d8695da3 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:24.160421 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af29e8fc-80dc-4def-ba9f-d49ffd6f1e14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:24.161678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af29e8fc-80dc-4def-ba9f-d49ffd6f1e14 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:24.161999 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af29e8fc-80dc-4def-ba9f-d49ffd6f1e14 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:24.162442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af29e8fc-80dc-4def-ba9f-d49ffd6f1e14 None admin] PATH_INFO: `/v3/users/1329a7ba9b3d48149f5a05ddeaf4bd31` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:24.163410 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af29e8fc-80dc-4def-ba9f-d49ffd6f1e14 None admin] RBAC: Authorizing `identity:get_user(user_id=1329a7ba9b3d48149f5a05ddeaf4bd31)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:24.164905 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af29e8fc-80dc-4def-ba9f-d49ffd6f1e14 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:24.170472 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 100/198] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:23 2022] GET /identity/v3/users/1329a7ba9b3d48149f5a05ddeaf4bd31 => generated 283 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:24.170472 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af29e8fc-80dc-4def-ba9f-d49ffd6f1e14 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:24.186507 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 99/199] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 404 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:06:24.188633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7220b24-489a-47b9-83a9-a62ae051810a None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:24.207732 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:24.292149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af29e8fc-80dc-4def-ba9f-d49ffd6f1e14 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:24.312418 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7220b24-489a-47b9-83a9-a62ae051810a None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:24.328605 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bbfa732-ed4f-4f31-b4b7-932d03295f12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:24.329430 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbfa732-ed4f-4f31-b4b7-932d03295f12 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:24.329720 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbfa732-ed4f-4f31-b4b7-932d03295f12 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:24.330024 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbfa732-ed4f-4f31-b4b7-932d03295f12 None admin] PATH_INFO: `/v3/users/71826307b6884fe78abc8b76724601d4` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:24.331856 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bbfa732-ed4f-4f31-b4b7-932d03295f12 None admin] RBAC: Authorizing `identity:get_user(user_id=71826307b6884fe78abc8b76724601d4)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:24.333858 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bbfa732-ed4f-4f31-b4b7-932d03295f12 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:24.335785 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 101/200] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:24 2022] GET /identity/v3/users/71826307b6884fe78abc8b76724601d4 => generated 289 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:24.336608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bbfa732-ed4f-4f31-b4b7-932d03295f12 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:24.359151 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855ce8b5-d8ec-403d-b463-9dd55fd03ce7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:24.359976 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855ce8b5-d8ec-403d-b463-9dd55fd03ce7 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:24.360635 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855ce8b5-d8ec-403d-b463-9dd55fd03ce7 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:24.361007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855ce8b5-d8ec-403d-b463-9dd55fd03ce7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:24.370139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:24.446705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-855ce8b5-d8ec-403d-b463-9dd55fd03ce7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:24.448744 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-855ce8b5-d8ec-403d-b463-9dd55fd03ce7 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:24.460554 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bbfa732-ed4f-4f31-b4b7-932d03295f12 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:24.473432 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:24.496685 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f1a982-e90a-494e-be85-3967362c0ff5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:24.497428 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f1a982-e90a-494e-be85-3967362c0ff5 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:24.497804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f1a982-e90a-494e-be85-3967362c0ff5 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:24.498166 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f1a982-e90a-494e-be85-3967362c0ff5 None admin] PATH_INFO: `/v3/projects/bbe6cda6a9564434b77be066e776bd55` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:24.499420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f1a982-e90a-494e-be85-3967362c0ff5 None admin] RBAC: Authorizing `identity:get_project(project_id=bbe6cda6a9564434b77be066e776bd55)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:24.500938 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f1a982-e90a-494e-be85-3967362c0ff5 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:24.507747 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 102/201] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:24 2022] GET /identity/v3/projects/bbe6cda6a9564434b77be066e776bd55 => generated 302 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:24.507747 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f1a982-e90a-494e-be85-3967362c0ff5 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:24.590354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:24.594992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 100/202] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:06:24.595927 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855ce8b5-d8ec-403d-b463-9dd55fd03ce7 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:24.621272 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f1a982-e90a-494e-be85-3967362c0ff5 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:24.670822 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fae9b7-4ab6-45f5-9753-b49ca2e67e45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:24.671593 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fae9b7-4ab6-45f5-9753-b49ca2e67e45 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:24.671968 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fae9b7-4ab6-45f5-9753-b49ca2e67e45 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:24.672337 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fae9b7-4ab6-45f5-9753-b49ca2e67e45 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:24.707591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855ce8b5-d8ec-403d-b463-9dd55fd03ce7 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:24.726425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:24.744538 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc479d7-870a-4738-a861-17c23f17de3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:24.745284 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc479d7-870a-4738-a861-17c23f17de3f None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:24.745703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc479d7-870a-4738-a861-17c23f17de3f None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:24.746096 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc479d7-870a-4738-a861-17c23f17de3f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:24.746915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc479d7-870a-4738-a861-17c23f17de3f None admin] RBAC: Adding query filter params (role.id=e6a31562a18649f0809202c36a8e18ef, scope.project.id=bbe6cda6a9564434b77be066e776bd55, user.id=1329a7ba9b3d48149f5a05ddeaf4bd31) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:24.747322 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc479d7-870a-4738-a861-17c23f17de3f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:24.748888 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc479d7-870a-4738-a861-17c23f17de3f None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:24.751229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33fae9b7-4ab6-45f5-9753-b49ca2e67e45 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:24.752959 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33fae9b7-4ab6-45f5-9753-b49ca2e67e45 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:24.755801 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 101/203] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:06:24 2022] GET /identity/v3/role_assignments?user.id=1329a7ba9b3d48149f5a05ddeaf4bd31&scope.project.id=bbe6cda6a9564434b77be066e776bd55&role.id=e6a31562a18649f0809202c36a8e18ef => generated 259 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:24.762942 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc479d7-870a-4738-a861-17c23f17de3f None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:24.846269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:24.884249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc479d7-870a-4738-a861-17c23f17de3f None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:24.894253 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 103/204] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:06:24.900323 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fae9b7-4ab6-45f5-9753-b49ca2e67e45 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:24.921899 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0d1ba8-3660-4c92-a378-a60e1729044b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:24.922542 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0d1ba8-3660-4c92-a378-a60e1729044b None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:24.922816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0d1ba8-3660-4c92-a378-a60e1729044b None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:24.923087 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0d1ba8-3660-4c92-a378-a60e1729044b None admin] PATH_INFO: `/v3/projects/bbe6cda6a9564434b77be066e776bd55` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:24.924221 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0d1ba8-3660-4c92-a378-a60e1729044b None admin] RBAC: Authorizing `identity:get_project(project_id=bbe6cda6a9564434b77be066e776bd55)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:24.925683 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0d1ba8-3660-4c92-a378-a60e1729044b None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:24.927641 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 102/205] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:24 2022] GET /identity/v3/projects/bbe6cda6a9564434b77be066e776bd55 => generated 302 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:24.933730 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0d1ba8-3660-4c92-a378-a60e1729044b None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:25.012988 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:25.015760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fae9b7-4ab6-45f5-9753-b49ca2e67e45 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:25.054696 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0e9f8a-3d85-464b-accf-3b3230606f52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:25.055428 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0e9f8a-3d85-464b-accf-3b3230606f52 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:25.055711 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0e9f8a-3d85-464b-accf-3b3230606f52 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:25.056014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0e9f8a-3d85-464b-accf-3b3230606f52 None admin] PATH_INFO: `/v3/projects/146fd1189c56413c9d17cd16d1a974d3` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:25.057197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0e9f8a-3d85-464b-accf-3b3230606f52 None admin] RBAC: Authorizing `identity:get_project(project_id=146fd1189c56413c9d17cd16d1a974d3)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:25.058688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0e9f8a-3d85-464b-accf-3b3230606f52 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:25.060572 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 104/206] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:24 2022] GET /identity/v3/projects/146fd1189c56413c9d17cd16d1a974d3 => generated 298 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:25.066406 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0e9f8a-3d85-464b-accf-3b3230606f52 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:25.067088 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0d1ba8-3660-4c92-a378-a60e1729044b None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:25.116444 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:25.134113 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6976a89f-e612-480a-9e66-f488e060768b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:25.134727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6976a89f-e612-480a-9e66-f488e060768b None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:25.135004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6976a89f-e612-480a-9e66-f488e060768b None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:25.135334 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6976a89f-e612-480a-9e66-f488e060768b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:25.136071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6976a89f-e612-480a-9e66-f488e060768b None admin] RBAC: Adding query filter params (role.id=2d9f391485604d0b952b7c844c3a3266, scope.project.id=bbe6cda6a9564434b77be066e776bd55, user.id=1329a7ba9b3d48149f5a05ddeaf4bd31) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:25.136374 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6976a89f-e612-480a-9e66-f488e060768b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:25.137834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6976a89f-e612-480a-9e66-f488e060768b None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:25.149043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 103/207] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:06:24 2022] GET /identity/v3/role_assignments?user.id=1329a7ba9b3d48149f5a05ddeaf4bd31&scope.project.id=bbe6cda6a9564434b77be066e776bd55&role.id=2d9f391485604d0b952b7c844c3a3266 => generated 259 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:25.176181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0e9f8a-3d85-464b-accf-3b3230606f52 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:25.215467 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6738c0da-1192-4ac1-9ff4-3495511b7794 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:25.216094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6738c0da-1192-4ac1-9ff4-3495511b7794 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:25.216430 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6738c0da-1192-4ac1-9ff4-3495511b7794 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:25.216705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6738c0da-1192-4ac1-9ff4-3495511b7794 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:25.218091 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6738c0da-1192-4ac1-9ff4-3495511b7794 None admin] RBAC: Adding query filter params (role.id=9b729a0f423e47f7bc562c000780ef7b, scope.project.id=146fd1189c56413c9d17cd16d1a974d3, user.id=71826307b6884fe78abc8b76724601d4) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:25.218514 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6738c0da-1192-4ac1-9ff4-3495511b7794 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:25.220013 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6738c0da-1192-4ac1-9ff4-3495511b7794 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:25.227335 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 105/208] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:06:25 2022] GET /identity/v3/role_assignments?user.id=71826307b6884fe78abc8b76724601d4&scope.project.id=146fd1189c56413c9d17cd16d1a974d3&role.id=9b729a0f423e47f7bc562c000780ef7b => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:26.256649 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f24f5d-7833-44eb-ac94-ef61bf68bb70 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:26.257175 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f24f5d-7833-44eb-ac94-ef61bf68bb70 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:26.257249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f24f5d-7833-44eb-ac94-ef61bf68bb70 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:26.258622 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 104/209] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:26.264107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31b246e-2f6a-400f-aeb4-e47bbefcdf79 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:26.264493 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31b246e-2f6a-400f-aeb4-e47bbefcdf79 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:26.264648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31b246e-2f6a-400f-aeb4-e47bbefcdf79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:26.283354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-e31b246e-2f6a-400f-aeb4-e47bbefcdf79 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:26.284536 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-e31b246e-2f6a-400f-aeb4-e47bbefcdf79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:26.292654 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810a4d4f-92ae-4a61-bbc0-c26a5928c810 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:26.292944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810a4d4f-92ae-4a61-bbc0-c26a5928c810 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:26.293226 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810a4d4f-92ae-4a61-bbc0-c26a5928c810 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:26.294589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 105/210] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:06:26.299529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d305257-288a-4375-99b8-d3b4419ba366 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:26.299815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d305257-288a-4375-99b8-d3b4419ba366 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:26.300095 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d305257-288a-4375-99b8-d3b4419ba366 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:26.318646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-2d305257-288a-4375-99b8-d3b4419ba366 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:26.319666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-2d305257-288a-4375-99b8-d3b4419ba366 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:26.355544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 106/211] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:06:26.387713 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 106/212] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:06:26.633398 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31b246e-2f6a-400f-aeb4-e47bbefcdf79 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:26.670482 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d305257-288a-4375-99b8-d3b4419ba366 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:26.680114 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:26.728931 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:26.750297 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31b246e-2f6a-400f-aeb4-e47bbefcdf79 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:26.788588 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d305257-288a-4375-99b8-d3b4419ba366 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:26.798659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ca02e0-0b36-452f-b340-acef305dc5cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:26.799971 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ca02e0-0b36-452f-b340-acef305dc5cf None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:26.800691 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ca02e0-0b36-452f-b340-acef305dc5cf None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:26.801138 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ca02e0-0b36-452f-b340-acef305dc5cf None admin] PATH_INFO: `/v3/roles/member` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:26.816263 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ca02e0-0b36-452f-b340-acef305dc5cf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:26.823312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-89ca02e0-0b36-452f-b340-acef305dc5cf None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:26.823312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:26.823312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:26.823312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:26.823312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:26.823312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:26.823312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:26.823312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:26.823312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:26.823312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:26.823312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:26.823312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:26.823312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:26.823312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:26.823312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:26.823312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:26.824174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:26.824174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:26.824174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:26.824174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:26.824174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:26.824174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:26.824174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:26.824174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:26.824174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:26.824174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:26.824174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:26.824174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:26.824174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:26.824174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:26.824174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:26.824174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:26.824950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:26.824950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:26.824950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:26.824950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:26.824950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:26.824950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:26.824950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:26.824950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:26.824950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:26.824950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:26.824950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:26.824950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:06:26.825539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c08d3f4-e103-4cd4-8c14-1fb62353fc44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:26.825539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c08d3f4-e103-4cd4-8c14-1fb62353fc44 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:26.825539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c08d3f4-e103-4cd4-8c14-1fb62353fc44 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:26.825539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c08d3f4-e103-4cd4-8c14-1fb62353fc44 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:26.827071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c08d3f4-e103-4cd4-8c14-1fb62353fc44 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:26.827823 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 107/213] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:26.833147 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c08d3f4-e103-4cd4-8c14-1fb62353fc44 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:26.834570 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ca02e0-0b36-452f-b340-acef305dc5cf None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:26.837776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 107/214] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:06:26 2022] GET /identity/v3/domains/default => generated 199 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:26.843148 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c08d3f4-e103-4cd4-8c14-1fb62353fc44 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:26.941405 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:26.946195 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ca02e0-0b36-452f-b340-acef305dc5cf None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:26.979256 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c08d3f4-e103-4cd4-8c14-1fb62353fc44 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:26.982748 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f344194-fc43-4332-9b35-8a068138a453 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:26.983517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f344194-fc43-4332-9b35-8a068138a453 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:26.983932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f344194-fc43-4332-9b35-8a068138a453 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:26.984420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f344194-fc43-4332-9b35-8a068138a453 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:26.985223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f344194-fc43-4332-9b35-8a068138a453 None admin] RBAC: Adding query filter params (name=member) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:26.985660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f344194-fc43-4332-9b35-8a068138a453 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:26.987172 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f344194-fc43-4332-9b35-8a068138a453 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:26.995816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:26.997390 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 108/215] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:26 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:27.003877 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f344194-fc43-4332-9b35-8a068138a453 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:27.024298 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46a30a5-6e71-453e-94dd-05640575b4fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:27.024936 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46a30a5-6e71-453e-94dd-05640575b4fd None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:27.025213 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46a30a5-6e71-453e-94dd-05640575b4fd None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:27.025476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46a30a5-6e71-453e-94dd-05640575b4fd None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:27.026171 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46a30a5-6e71-453e-94dd-05640575b4fd None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:27.027950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46a30a5-6e71-453e-94dd-05640575b4fd None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:27.061703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 108/216] 10.222.0.26 () {60 vars in 1132 bytes} [Tue Sep 27 04:06:26 2022] POST /identity/v3/users => generated 297 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:06:27.133650 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f344194-fc43-4332-9b35-8a068138a453 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:27.172953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6eed7a-1a2d-40bc-844f-506c47613b35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:27.173591 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6eed7a-1a2d-40bc-844f-506c47613b35 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:27.173906 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6eed7a-1a2d-40bc-844f-506c47613b35 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:27.174176 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6eed7a-1a2d-40bc-844f-506c47613b35 None admin] PATH_INFO: `/v3/users/4cc4e06ab0b441e785c85a4107e0dcae` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:27.189942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d6eed7a-1a2d-40bc-844f-506c47613b35 None admin] RBAC: Authorizing `identity:get_user(user_id=4cc4e06ab0b441e785c85a4107e0dcae)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:27.191505 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d6eed7a-1a2d-40bc-844f-506c47613b35 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:27.193950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 109/217] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:27 2022] GET /identity/v3/users/4cc4e06ab0b441e785c85a4107e0dcae => generated 297 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:27.203286 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46a30a5-6e71-453e-94dd-05640575b4fd None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:27.323286 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46a30a5-6e71-453e-94dd-05640575b4fd None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:27.357059 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63128ce4-79fb-496a-bde6-465f122fe013 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:27.357572 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63128ce4-79fb-496a-bde6-465f122fe013 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:27.357572 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63128ce4-79fb-496a-bde6-465f122fe013 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:27.359056 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 110/218] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:06:27.359226 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745f0aab-4898-4916-9c00-d3fac93c4ce9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:27.359837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745f0aab-4898-4916-9c00-d3fac93c4ce9 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:27.360112 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745f0aab-4898-4916-9c00-d3fac93c4ce9 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:27.360379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745f0aab-4898-4916-9c00-d3fac93c4ce9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:27.366882 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75371c86-fffe-499c-9087-fd64790e2849 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:27.367162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75371c86-fffe-499c-9087-fd64790e2849 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:27.367431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75371c86-fffe-499c-9087-fd64790e2849 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:27.396649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-75371c86-fffe-499c-9087-fd64790e2849 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:27.399148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-75371c86-fffe-499c-9087-fd64790e2849 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:27.407673 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:27.429579 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:27.450067 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745f0aab-4898-4916-9c00-d3fac93c4ce9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:27.451956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745f0aab-4898-4916-9c00-d3fac93c4ce9 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:27.503022 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 111/219] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:27.526009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe60611-cf80-4ae3-90e2-c3dd6bad0b01 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:27.526434 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe60611-cf80-4ae3-90e2-c3dd6bad0b01 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:27.526813 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe60611-cf80-4ae3-90e2-c3dd6bad0b01 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:27.529261 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 112/220] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:06:27.534410 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3727e722-bb3f-44e3-b8e7-788efb9a058c None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:27.534803 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3727e722-bb3f-44e3-b8e7-788efb9a058c None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:27.535179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3727e722-bb3f-44e3-b8e7-788efb9a058c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:27.560674 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-3727e722-bb3f-44e3-b8e7-788efb9a058c None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:27.562056 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-3727e722-bb3f-44e3-b8e7-788efb9a058c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:27.605769 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 109/221] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:06:27.606671 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745f0aab-4898-4916-9c00-d3fac93c4ce9 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:27.673122 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 113/222] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:06:27.711588 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:27.747086 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745f0aab-4898-4916-9c00-d3fac93c4ce9 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:27.781968 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d60ab6-d233-46cb-af56-d2553bddcec7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:27.782608 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d60ab6-d233-46cb-af56-d2553bddcec7 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:27.782885 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d60ab6-d233-46cb-af56-d2553bddcec7 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:27.783215 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d60ab6-d233-46cb-af56-d2553bddcec7 None admin] PATH_INFO: `/v3/projects/bbe6cda6a9564434b77be066e776bd55` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:27.784402 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d60ab6-d233-46cb-af56-d2553bddcec7 None admin] RBAC: Authorizing `identity:get_project(project_id=bbe6cda6a9564434b77be066e776bd55)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:27.785924 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d60ab6-d233-46cb-af56-d2553bddcec7 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:27.788381 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 110/223] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:27 2022] GET /identity/v3/projects/bbe6cda6a9564434b77be066e776bd55 => generated 302 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:27.794637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3727e722-bb3f-44e3-b8e7-788efb9a058c None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:27.816390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d60ab6-d233-46cb-af56-d2553bddcec7 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:27.937770 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:27.957128 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3727e722-bb3f-44e3-b8e7-788efb9a058c None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:27.982274 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d60ab6-d233-46cb-af56-d2553bddcec7 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:27.996034 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f544a79a-123d-4a86-8acf-b00d635be803 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:27.996904 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f544a79a-123d-4a86-8acf-b00d635be803 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:27.997268 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f544a79a-123d-4a86-8acf-b00d635be803 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:27.997721 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f544a79a-123d-4a86-8acf-b00d635be803 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:27.998598 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f544a79a-123d-4a86-8acf-b00d635be803 None admin] RBAC: Adding query filter params (role.id=e3ce5a3697ec4292a446366d3caadeea, scope.project.id=bbe6cda6a9564434b77be066e776bd55, user.id=4cc4e06ab0b441e785c85a4107e0dcae) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:27.998969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f544a79a-123d-4a86-8acf-b00d635be803 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:28.000738 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f544a79a-123d-4a86-8acf-b00d635be803 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:28.008767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 114/224] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:06:27 2022] GET /identity/v3/role_assignments?user.id=4cc4e06ab0b441e785c85a4107e0dcae&scope.project.id=bbe6cda6a9564434b77be066e776bd55&role.id=e3ce5a3697ec4292a446366d3caadeea => generated 259 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:28.009573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f544a79a-123d-4a86-8acf-b00d635be803 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:28.037826 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227705b3-6312-4287-a60b-f1d44ba01b46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:28.038494 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227705b3-6312-4287-a60b-f1d44ba01b46 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:28.038771 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227705b3-6312-4287-a60b-f1d44ba01b46 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:28.039040 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227705b3-6312-4287-a60b-f1d44ba01b46 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:28.043347 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:28.062434 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227705b3-6312-4287-a60b-f1d44ba01b46 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:28.067685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-227705b3-6312-4287-a60b-f1d44ba01b46 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:06:28.067685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:28.067685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:28.067685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:28.067685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:28.067685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:28.067685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:28.067685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:28.067685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:28.067685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:28.067685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:28.067685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:28.067685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:28.067685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:28.067685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:28.067685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:28.068580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:28.068580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:28.068580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:28.068580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:28.068580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:28.068580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:28.068580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:28.068580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:28.068580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:28.068580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:28.068580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:28.068580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:28.068580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:28.068580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:28.068580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:28.068580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:28.069339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:28.069339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:28.069339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:28.069339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:28.069339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:28.069339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:28.069339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:28.069339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:28.069339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:28.069339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:28.069339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:06:28.069339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:06:28.069958 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 111/225] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:06:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:28.077297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227705b3-6312-4287-a60b-f1d44ba01b46 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:28.243003 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f544a79a-123d-4a86-8acf-b00d635be803 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:28.254294 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227705b3-6312-4287-a60b-f1d44ba01b46 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:28.301009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d68d5db-c115-4171-af00-f80240ef814d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:28.301893 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d68d5db-c115-4171-af00-f80240ef814d None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:28.302172 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d68d5db-c115-4171-af00-f80240ef814d None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:28.303239 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d68d5db-c115-4171-af00-f80240ef814d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:28.318975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542442f8-abb3-4068-9236-8a923f4e9fb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:28.319582 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542442f8-abb3-4068-9236-8a923f4e9fb7 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:28.319856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542442f8-abb3-4068-9236-8a923f4e9fb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:28.320126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542442f8-abb3-4068-9236-8a923f4e9fb7 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:28.320799 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542442f8-abb3-4068-9236-8a923f4e9fb7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:28.321093 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542442f8-abb3-4068-9236-8a923f4e9fb7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:28.326643 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542442f8-abb3-4068-9236-8a923f4e9fb7 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:28.328680 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d68d5db-c115-4171-af00-f80240ef814d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:28.334700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-1d68d5db-c115-4171-af00-f80240ef814d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:06:28.334700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:28.334700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:28.334700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:28.334700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:28.334700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:28.334700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:28.334700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:28.334700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:28.334700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:28.334700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:28.334700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:28.334700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:28.334700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:28.334700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:28.334700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:28.335768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:28.335768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:28.335768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:28.335768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:28.335768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:28.335768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:28.335768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:28.335768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:28.335768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:28.335768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:28.335768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:28.335768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:28.335768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:28.335768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:28.335768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:28.335768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:28.336739 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:28.336739 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:28.336739 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:28.336739 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:28.336739 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:28.336739 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:28.336739 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:28.336739 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:28.336739 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:28.336739 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:28.336739 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:06:28.336739 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:06:28.339126 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 115/226] 10.222.0.26 () {56 vars in 1114 bytes} [Tue Sep 27 04:06:28 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 330 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:28.346881 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d68d5db-c115-4171-af00-f80240ef814d None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:28.352066 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 112/227] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:28.352491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:28.359686 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542442f8-abb3-4068-9236-8a923f4e9fb7 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:28.468356 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:28.537672 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d68d5db-c115-4171-af00-f80240ef814d None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:28.549190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542442f8-abb3-4068-9236-8a923f4e9fb7 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:28.599005 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c42c8b-ed17-47f0-8b85-6c86ed612d92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:28.599623 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c42c8b-ed17-47f0-8b85-6c86ed612d92 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:28.599903 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c42c8b-ed17-47f0-8b85-6c86ed612d92 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:28.600173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c42c8b-ed17-47f0-8b85-6c86ed612d92 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:28.600839 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c42c8b-ed17-47f0-8b85-6c86ed612d92 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:28.601137 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c42c8b-ed17-47f0-8b85-6c86ed612d92 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:28.602911 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c42c8b-ed17-47f0-8b85-6c86ed612d92 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:28.611570 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 116/228] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:06:28 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:28.617131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c42c8b-ed17-47f0-8b85-6c86ed612d92 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:28.620793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30934be-c5df-419d-9710-6b92e15e0198 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:28.621963 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30934be-c5df-419d-9710-6b92e15e0198 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:28.621963 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30934be-c5df-419d-9710-6b92e15e0198 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:28.621963 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30934be-c5df-419d-9710-6b92e15e0198 None admin] PATH_INFO: `/v3/users/1329a7ba9b3d48149f5a05ddeaf4bd31` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:28.623146 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30934be-c5df-419d-9710-6b92e15e0198 None admin] RBAC: Authorizing `identity:get_user(user_id=1329a7ba9b3d48149f5a05ddeaf4bd31)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:28.624590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30934be-c5df-419d-9710-6b92e15e0198 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:28.626222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 113/229] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:28 2022] GET /identity/v3/users/1329a7ba9b3d48149f5a05ddeaf4bd31 => generated 283 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:28.632477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30934be-c5df-419d-9710-6b92e15e0198 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:28.698513 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:28.739425 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c42c8b-ed17-47f0-8b85-6c86ed612d92 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:28.786027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91a0454-e80c-47bf-bf5b-2915483e5274 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:28.786998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91a0454-e80c-47bf-bf5b-2915483e5274 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:28.787555 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91a0454-e80c-47bf-bf5b-2915483e5274 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:28.788056 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91a0454-e80c-47bf-bf5b-2915483e5274 None admin] PATH_INFO: `/v3/users/1329a7ba9b3d48149f5a05ddeaf4bd31` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:28.790490 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91a0454-e80c-47bf-bf5b-2915483e5274 None admin] RBAC: Authorizing `identity:get_user(user_id=1329a7ba9b3d48149f5a05ddeaf4bd31)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:28.792103 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91a0454-e80c-47bf-bf5b-2915483e5274 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:28.796085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 117/230] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:28 2022] GET /identity/v3/users/1329a7ba9b3d48149f5a05ddeaf4bd31 => generated 283 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:28.802931 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91a0454-e80c-47bf-bf5b-2915483e5274 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:28.843650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30934be-c5df-419d-9710-6b92e15e0198 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:28.910319 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91a0454-e80c-47bf-bf5b-2915483e5274 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:28.935319 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:28.947239 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871dedb5-faa1-4e6b-a07b-a82f90af51ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:28.948434 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871dedb5-faa1-4e6b-a07b-a82f90af51ff None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:28.948891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871dedb5-faa1-4e6b-a07b-a82f90af51ff None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:28.949346 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871dedb5-faa1-4e6b-a07b-a82f90af51ff None admin] PATH_INFO: `/v3/projects/bbe6cda6a9564434b77be066e776bd55` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:28.950938 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871dedb5-faa1-4e6b-a07b-a82f90af51ff None admin] RBAC: Authorizing `identity:get_project(project_id=bbe6cda6a9564434b77be066e776bd55)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:28.952544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871dedb5-faa1-4e6b-a07b-a82f90af51ff None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:28.954727 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 118/231] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:28 2022] GET /identity/v3/projects/bbe6cda6a9564434b77be066e776bd55 => generated 302 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:28.962936 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871dedb5-faa1-4e6b-a07b-a82f90af51ff None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:28.967547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:28.970447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cce321-f9e1-4c1e-8a7f-08b050b6f5db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:28.971104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cce321-f9e1-4c1e-8a7f-08b050b6f5db None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:28.971384 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cce321-f9e1-4c1e-8a7f-08b050b6f5db None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:28.971658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cce321-f9e1-4c1e-8a7f-08b050b6f5db None admin] PATH_INFO: `/v3/projects/bbe6cda6a9564434b77be066e776bd55` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:28.972949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50cce321-f9e1-4c1e-8a7f-08b050b6f5db None admin] RBAC: Authorizing `identity:get_project(project_id=bbe6cda6a9564434b77be066e776bd55)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:28.974897 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50cce321-f9e1-4c1e-8a7f-08b050b6f5db None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:28.976677 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 114/232] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:28 2022] GET /identity/v3/projects/bbe6cda6a9564434b77be066e776bd55 => generated 302 bytes in 344 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:28.981980 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cce321-f9e1-4c1e-8a7f-08b050b6f5db None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:29.076496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871dedb5-faa1-4e6b-a07b-a82f90af51ff None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:29.113641 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0933a4-ccb7-4a98-8aa3-0714a8233501 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:29.115186 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0933a4-ccb7-4a98-8aa3-0714a8233501 None admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:29.115686 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0933a4-ccb7-4a98-8aa3-0714a8233501 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:29.116545 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0933a4-ccb7-4a98-8aa3-0714a8233501 None admin] PATH_INFO: `/v3/projects/bbe6cda6a9564434b77be066e776bd55/users/1329a7ba9b3d48149f5a05ddeaf4bd31/roles/2d9f391485604d0b952b7c844c3a3266` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:29.119291 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0933a4-ccb7-4a98-8aa3-0714a8233501 None admin] RBAC: Authorizing `identity:create_grant(project_id=bbe6cda6a9564434b77be066e776bd55, user_id=1329a7ba9b3d48149f5a05ddeaf4bd31, role_id=2d9f391485604d0b952b7c844c3a3266)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:29.120929 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0933a4-ccb7-4a98-8aa3-0714a8233501 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:29.137270 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 119/233] 10.222.0.26 () {58 vars in 1439 bytes} [Tue Sep 27 04:06:28 2022] PUT /identity/v3/projects/bbe6cda6a9564434b77be066e776bd55/users/1329a7ba9b3d48149f5a05ddeaf4bd31/roles/2d9f391485604d0b952b7c844c3a3266 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:06:29.202600 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cce321-f9e1-4c1e-8a7f-08b050b6f5db None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:29.248765 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34876e3-812c-4694-8a28-b35e19fac0ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:29.249456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34876e3-812c-4694-8a28-b35e19fac0ee None admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:29.249727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34876e3-812c-4694-8a28-b35e19fac0ee None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:29.250022 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34876e3-812c-4694-8a28-b35e19fac0ee None admin] PATH_INFO: `/v3/projects/bbe6cda6a9564434b77be066e776bd55/users/1329a7ba9b3d48149f5a05ddeaf4bd31/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:29.254465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34876e3-812c-4694-8a28-b35e19fac0ee None admin] RBAC: Authorizing `identity:create_grant(project_id=bbe6cda6a9564434b77be066e776bd55, user_id=1329a7ba9b3d48149f5a05ddeaf4bd31, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:29.256022 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34876e3-812c-4694-8a28-b35e19fac0ee None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:29.283864 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 115/234] 10.222.0.26 () {58 vars in 1439 bytes} [Tue Sep 27 04:06:28 2022] PUT /identity/v3/projects/bbe6cda6a9564434b77be066e776bd55/users/1329a7ba9b3d48149f5a05ddeaf4bd31/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:06:29.727851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3b0247-e6c7-4e57-a3d9-f4fb8bcb1875 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:29.728828 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3b0247-e6c7-4e57-a3d9-f4fb8bcb1875 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:29.729333 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3b0247-e6c7-4e57-a3d9-f4fb8bcb1875 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:29.731346 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 120/235] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:29.737265 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1869257-5b30-4cb2-bb27-2ee417b90882 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:29.738149 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1869257-5b30-4cb2-bb27-2ee417b90882 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:29.738803 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1869257-5b30-4cb2-bb27-2ee417b90882 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:29.758186 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-c1869257-5b30-4cb2-bb27-2ee417b90882 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:29.759676 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c1869257-5b30-4cb2-bb27-2ee417b90882 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:29.831737 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:29.856699 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 116/236] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:29.890717 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e031bac5-ef5f-489c-84e5-270cf7cf8ab3 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:29.891315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e031bac5-ef5f-489c-84e5-270cf7cf8ab3 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:29.891809 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e031bac5-ef5f-489c-84e5-270cf7cf8ab3 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:29.893658 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 121/237] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:29.899212 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7627dfd4-1d80-4e10-b972-5efea31637df None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:29.899712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7627dfd4-1d80-4e10-b972-5efea31637df None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:29.900196 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7627dfd4-1d80-4e10-b972-5efea31637df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:29.922600 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-7627dfd4-1d80-4e10-b972-5efea31637df None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:29.924185 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-7627dfd4-1d80-4e10-b972-5efea31637df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:29.991677 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 117/238] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:30.206609 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e031bac5-ef5f-489c-84e5-270cf7cf8ab3 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:30.273754 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:30.316648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e031bac5-ef5f-489c-84e5-270cf7cf8ab3 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:30.351744 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03af10d9-bff6-4a36-a49f-2f3ebefaeccc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:30.352371 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03af10d9-bff6-4a36-a49f-2f3ebefaeccc None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:30.352685 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03af10d9-bff6-4a36-a49f-2f3ebefaeccc None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:30.352966 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03af10d9-bff6-4a36-a49f-2f3ebefaeccc None admin] PATH_INFO: `/v3/domains/default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:30.354093 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03af10d9-bff6-4a36-a49f-2f3ebefaeccc None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:30.355532 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03af10d9-bff6-4a36-a49f-2f3ebefaeccc None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:30.357441 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 122/239] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:06:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:30.363086 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7627dfd4-1d80-4e10-b972-5efea31637df None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:30.429835 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03af10d9-bff6-4a36-a49f-2f3ebefaeccc None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:30.512806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7627dfd4-1d80-4e10-b972-5efea31637df None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:30.539560 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03af10d9-bff6-4a36-a49f-2f3ebefaeccc None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:30.540934 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:30.580236 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0f980a-2149-4f56-bb6a-67dec334b5a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:30.581205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0f980a-2149-4f56-bb6a-67dec334b5a6 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:30.581947 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0f980a-2149-4f56-bb6a-67dec334b5a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:30.582654 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0f980a-2149-4f56-bb6a-67dec334b5a6 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:30.586452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2881a5e-8f23-4970-92ee-649a2ef1c42a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:30.587325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2881a5e-8f23-4970-92ee-649a2ef1c42a None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:30.587325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2881a5e-8f23-4970-92ee-649a2ef1c42a None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:30.587603 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2881a5e-8f23-4970-92ee-649a2ef1c42a None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:30.588258 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2881a5e-8f23-4970-92ee-649a2ef1c42a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:30.589707 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2881a5e-8f23-4970-92ee-649a2ef1c42a None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:30.597292 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:30.597787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0f980a-2149-4f56-bb6a-67dec334b5a6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:30.604326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-ed0f980a-2149-4f56-bb6a-67dec334b5a6 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 04:06:30.604326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:30.604326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:30.604326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:30.604326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:30.604326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:30.604326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:30.604326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:30.604326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:30.604326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:30.604326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:30.604326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:30.604326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:30.604326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:30.604326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:30.604326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:30.606050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:30.606050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:30.606050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:30.606050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:30.606050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:30.606050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:30.606050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:30.606050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:30.606050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:30.606050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:30.606050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:30.606050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:30.606050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:30.606050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:30.606050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:30.606050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:30.607690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:30.607690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:30.607690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:30.607690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:30.607690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:30.607690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:30.607690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:30.607690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:30.607690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:30.607690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:30.607690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 04:06:30.607690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:06:30.610792 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 123/240] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:30 2022] GET /identity/v3/roles/reader => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:06:30.617779 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0f980a-2149-4f56-bb6a-67dec334b5a6 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:30.632974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 118/241] 10.222.0.26 () {60 vars in 1132 bytes} [Tue Sep 27 04:06:30 2022] POST /identity/v3/users => generated 293 bytes in 270 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:06:30.799164 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540cfc52-580e-4a44-963b-fe3f61b7b19a None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:30.799995 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540cfc52-580e-4a44-963b-fe3f61b7b19a None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:30.799995 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540cfc52-580e-4a44-963b-fe3f61b7b19a None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:30.801173 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 119/242] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:06:30.815297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e650429-1a75-4049-849d-a70bd7787d89 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:30.815630 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e650429-1a75-4049-849d-a70bd7787d89 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:30.815895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e650429-1a75-4049-849d-a70bd7787d89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:30.818645 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0f980a-2149-4f56-bb6a-67dec334b5a6 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:30.855823 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-0e650429-1a75-4049-849d-a70bd7787d89 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:30.856774 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-0e650429-1a75-4049-849d-a70bd7787d89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:30.872385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a646fa5b-a2fa-43fd-b8f1-3e47d9ac80bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:30.873386 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a646fa5b-a2fa-43fd-b8f1-3e47d9ac80bb None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:30.873910 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a646fa5b-a2fa-43fd-b8f1-3e47d9ac80bb None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:30.874474 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a646fa5b-a2fa-43fd-b8f1-3e47d9ac80bb None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:30.875374 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a646fa5b-a2fa-43fd-b8f1-3e47d9ac80bb None admin] RBAC: Adding query filter params (name=reader) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:30.875909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a646fa5b-a2fa-43fd-b8f1-3e47d9ac80bb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:30.877874 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a646fa5b-a2fa-43fd-b8f1-3e47d9ac80bb None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:30.890359 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 124/243] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:30 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:30.897888 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a646fa5b-a2fa-43fd-b8f1-3e47d9ac80bb None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:30.998152 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:31.003180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 120/244] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:31.041511 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a646fa5b-a2fa-43fd-b8f1-3e47d9ac80bb None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:31.086376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1c9084-b5c8-43c3-912e-3132c67c5468 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:31.087454 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1c9084-b5c8-43c3-912e-3132c67c5468 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:31.088207 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1c9084-b5c8-43c3-912e-3132c67c5468 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:31.088949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1c9084-b5c8-43c3-912e-3132c67c5468 None admin] PATH_INFO: `/v3/users/495345eb577b47a79c3fda23f8bc23eb` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:31.108281 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1c9084-b5c8-43c3-912e-3132c67c5468 None admin] RBAC: Authorizing `identity:get_user(user_id=495345eb577b47a79c3fda23f8bc23eb)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:31.110128 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1c9084-b5c8-43c3-912e-3132c67c5468 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:31.113167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 125/245] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:30 2022] GET /identity/v3/users/495345eb577b47a79c3fda23f8bc23eb => generated 297 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:31.119662 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e650429-1a75-4049-849d-a70bd7787d89 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:31.238072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:31.244146 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e650429-1a75-4049-849d-a70bd7787d89 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:31.287462 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef98a15-9714-4990-a938-5768a6ec71be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:31.288081 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef98a15-9714-4990-a938-5768a6ec71be None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:31.288355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef98a15-9714-4990-a938-5768a6ec71be None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:31.288624 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef98a15-9714-4990-a938-5768a6ec71be None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:31.321669 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1c9084-b5c8-43c3-912e-3132c67c5468 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:31.394214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef98a15-9714-4990-a938-5768a6ec71be None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:31.396904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef98a15-9714-4990-a938-5768a6ec71be None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:31.493703 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1c9084-b5c8-43c3-912e-3132c67c5468 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:31.511246 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:31.537533 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:31.538615 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc4a438-4010-4769-8f54-88eee18da714 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:31.539274 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc4a438-4010-4769-8f54-88eee18da714 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:31.540301 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc4a438-4010-4769-8f54-88eee18da714 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:31.540578 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc4a438-4010-4769-8f54-88eee18da714 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:31.543665 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 121/246] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 424 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:06:31.551529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef98a15-9714-4990-a938-5768a6ec71be None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:31.557024 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc4a438-4010-4769-8f54-88eee18da714 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:31.562849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-7bc4a438-4010-4769-8f54-88eee18da714 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:31.562849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:31.562849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:31.562849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:31.562849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:31.562849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:31.562849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:31.562849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:31.562849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:31.562849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:31.562849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:31.562849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:31.562849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:31.562849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:31.562849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:31.562849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:31.563926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:31.563926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:31.563926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:31.563926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:31.563926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:31.563926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:31.563926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:31.563926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:31.563926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:31.563926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:31.563926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:31.563926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:31.563926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:31.563926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:31.563926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:31.563926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:31.565062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:31.565062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:31.565062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:31.565062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:31.565062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:31.565062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:31.565062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:31.565062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:31.565062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:31.565062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:31.565062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:31.565062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:06:31.565062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 126/247] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:31 2022] GET /identity/v3/roles/member => generated 84 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:06:31.570121 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc4a438-4010-4769-8f54-88eee18da714 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:31.685168 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef98a15-9714-4990-a938-5768a6ec71be None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:31.701140 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc4a438-4010-4769-8f54-88eee18da714 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:31.730532 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99355013-64fc-4980-b0dc-a4d9828d146f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:31.731484 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99355013-64fc-4980-b0dc-a4d9828d146f None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:31.731921 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99355013-64fc-4980-b0dc-a4d9828d146f None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:31.732326 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99355013-64fc-4980-b0dc-a4d9828d146f None admin] PATH_INFO: `/v3/projects/bbe6cda6a9564434b77be066e776bd55` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:31.738551 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99355013-64fc-4980-b0dc-a4d9828d146f None admin] RBAC: Authorizing `identity:get_project(project_id=bbe6cda6a9564434b77be066e776bd55)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:31.740222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99355013-64fc-4980-b0dc-a4d9828d146f None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:31.742181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 122/248] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:31 2022] GET /identity/v3/projects/bbe6cda6a9564434b77be066e776bd55 => generated 302 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:31.747967 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99355013-64fc-4980-b0dc-a4d9828d146f None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:31.775632 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4791fd-36ef-4066-abf7-d53eb5e9b8fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:31.776259 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4791fd-36ef-4066-abf7-d53eb5e9b8fa None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:31.776538 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4791fd-36ef-4066-abf7-d53eb5e9b8fa None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:31.776810 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4791fd-36ef-4066-abf7-d53eb5e9b8fa None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:31.780563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4791fd-36ef-4066-abf7-d53eb5e9b8fa None admin] RBAC: Adding query filter params (name=member) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:31.780563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4791fd-36ef-4066-abf7-d53eb5e9b8fa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:31.780563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4791fd-36ef-4066-abf7-d53eb5e9b8fa None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:31.793679 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 127/249] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:31 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:31.803325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4791fd-36ef-4066-abf7-d53eb5e9b8fa None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:31.850444 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:31.862103 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:31.915445 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99355013-64fc-4980-b0dc-a4d9828d146f None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:31.974594 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16172b9b-ea0b-4b1d-8ab9-14a6e825f17e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:31.975131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16172b9b-ea0b-4b1d-8ab9-14a6e825f17e None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:31.975414 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16172b9b-ea0b-4b1d-8ab9-14a6e825f17e None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:31.975686 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16172b9b-ea0b-4b1d-8ab9-14a6e825f17e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:31.976426 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16172b9b-ea0b-4b1d-8ab9-14a6e825f17e None admin] RBAC: Adding query filter params (role.id=9b729a0f423e47f7bc562c000780ef7b, scope.project.id=bbe6cda6a9564434b77be066e776bd55, user.id=495345eb577b47a79c3fda23f8bc23eb) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:31.976733 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16172b9b-ea0b-4b1d-8ab9-14a6e825f17e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:31.978364 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16172b9b-ea0b-4b1d-8ab9-14a6e825f17e None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:31.984487 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 123/250] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:06:31 2022] GET /identity/v3/role_assignments?user.id=495345eb577b47a79c3fda23f8bc23eb&scope.project.id=bbe6cda6a9564434b77be066e776bd55&role.id=9b729a0f423e47f7bc562c000780ef7b => generated 259 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:31.989102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6ff9a6-fafe-48f7-a0a1-3f4ab4dd6c6a None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:31.989102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6ff9a6-fafe-48f7-a0a1-3f4ab4dd6c6a None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:31.989102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6ff9a6-fafe-48f7-a0a1-3f4ab4dd6c6a None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:31.989102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 124/251] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:31.994527 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe10489-9541-446e-b55c-5c1b77d3cc0c None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:31.994810 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe10489-9541-446e-b55c-5c1b77d3cc0c None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:31.995080 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe10489-9541-446e-b55c-5c1b77d3cc0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:32.004430 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4791fd-36ef-4066-abf7-d53eb5e9b8fa None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:32.026889 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-0fe10489-9541-446e-b55c-5c1b77d3cc0c None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:32.028092 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-0fe10489-9541-446e-b55c-5c1b77d3cc0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:32.064728 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f133d1-7d81-44e9-b732-01d1cf136e5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:32.065834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f133d1-7d81-44e9-b732-01d1cf136e5e None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:32.066323 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f133d1-7d81-44e9-b732-01d1cf136e5e None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:32.066805 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f133d1-7d81-44e9-b732-01d1cf136e5e None admin] PATH_INFO: `/v3/users/4cc4e06ab0b441e785c85a4107e0dcae` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:32.074329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f133d1-7d81-44e9-b732-01d1cf136e5e None admin] RBAC: Authorizing `identity:get_user(user_id=4cc4e06ab0b441e785c85a4107e0dcae)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:32.076363 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f133d1-7d81-44e9-b732-01d1cf136e5e None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:32.078743 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 128/252] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:31 2022] GET /identity/v3/users/4cc4e06ab0b441e785c85a4107e0dcae => generated 297 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:32.084291 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f133d1-7d81-44e9-b732-01d1cf136e5e None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:32.140157 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 125/253] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:06:32.181787 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a15f9b-e351-4255-b697-d20966a848fb None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:32.182089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a15f9b-e351-4255-b697-d20966a848fb None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:32.182341 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a15f9b-e351-4255-b697-d20966a848fb None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:32.184807 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 126/254] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:32.189143 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700761a6-7ecf-41cf-b969-4081edda8a76 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:32.189451 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700761a6-7ecf-41cf-b969-4081edda8a76 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:32.189702 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700761a6-7ecf-41cf-b969-4081edda8a76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:32.223172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-700761a6-7ecf-41cf-b969-4081edda8a76 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:32.226184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-700761a6-7ecf-41cf-b969-4081edda8a76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:32.235281 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:32.244552 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f133d1-7d81-44e9-b732-01d1cf136e5e None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:32.293331 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504647bf-d63e-4cf8-be63-a67fb63848c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:32.294340 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504647bf-d63e-4cf8-be63-a67fb63848c0 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:32.294837 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504647bf-d63e-4cf8-be63-a67fb63848c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:32.295298 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504647bf-d63e-4cf8-be63-a67fb63848c0 None admin] PATH_INFO: `/v3/projects/bbe6cda6a9564434b77be066e776bd55` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:32.301789 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504647bf-d63e-4cf8-be63-a67fb63848c0 None admin] RBAC: Authorizing `identity:get_project(project_id=bbe6cda6a9564434b77be066e776bd55)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:32.304615 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504647bf-d63e-4cf8-be63-a67fb63848c0 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:32.307397 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 129/255] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:32 2022] GET /identity/v3/projects/bbe6cda6a9564434b77be066e776bd55 => generated 302 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:32.312773 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504647bf-d63e-4cf8-be63-a67fb63848c0 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:32.319693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:32.338163 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 127/256] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:06:32.444818 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504647bf-d63e-4cf8-be63-a67fb63848c0 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:32.479254 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d6633b-8418-491d-86e2-18cc1e2ade0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:32.480236 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d6633b-8418-491d-86e2-18cc1e2ade0c None admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:32.480706 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d6633b-8418-491d-86e2-18cc1e2ade0c None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:32.481187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d6633b-8418-491d-86e2-18cc1e2ade0c None admin] PATH_INFO: `/v3/projects/bbe6cda6a9564434b77be066e776bd55/users/4cc4e06ab0b441e785c85a4107e0dcae/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:32.483299 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d6633b-8418-491d-86e2-18cc1e2ade0c None admin] RBAC: Authorizing `identity:create_grant(project_id=bbe6cda6a9564434b77be066e776bd55, user_id=4cc4e06ab0b441e785c85a4107e0dcae, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:32.484921 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d6633b-8418-491d-86e2-18cc1e2ade0c None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:32.507332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 130/257] 10.222.0.26 () {58 vars in 1439 bytes} [Tue Sep 27 04:06:32 2022] PUT /identity/v3/projects/bbe6cda6a9564434b77be066e776bd55/users/4cc4e06ab0b441e785c85a4107e0dcae/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:06:32.613227 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700761a6-7ecf-41cf-b969-4081edda8a76 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:32.739974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700761a6-7ecf-41cf-b969-4081edda8a76 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:32.780167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d6633b-8418-491d-86e2-18cc1e2ade0c None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:32.788909 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e2e5af-8bd4-4136-8033-02aabc25b299 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:32.789862 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e2e5af-8bd4-4136-8033-02aabc25b299 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:32.790335 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e2e5af-8bd4-4136-8033-02aabc25b299 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:32.790913 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e2e5af-8bd4-4136-8033-02aabc25b299 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:32.816000 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e2e5af-8bd4-4136-8033-02aabc25b299 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:32.823777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-95e2e5af-8bd4-4136-8033-02aabc25b299 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:06:32.823777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:32.823777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:32.823777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:32.823777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:32.823777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:32.823777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:32.823777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:32.823777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:32.823777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:32.823777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:32.823777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:32.823777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:32.823777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:32.823777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:32.823777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:32.824715 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:32.824715 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:32.824715 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:32.824715 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:32.824715 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:32.824715 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:32.824715 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:32.824715 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:32.824715 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:32.824715 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:32.824715 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:32.824715 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:32.824715 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:32.824715 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:32.824715 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:32.824715 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:32.825509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:32.825509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:32.825509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:32.825509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:32.825509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:32.825509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:32.825509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:32.825509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:32.825509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:32.825509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:32.825509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:06:32.825509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:06:32.828240 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 128/258] 10.222.0.26 () {56 vars in 1114 bytes} [Tue Sep 27 04:06:32 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:32.831918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:32.834549 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e2e5af-8bd4-4136-8033-02aabc25b299 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:32.926393 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:32.972049 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d6633b-8418-491d-86e2-18cc1e2ade0c None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:32.978261 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e2e5af-8bd4-4136-8033-02aabc25b299 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:33.028089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dffcb3-b7aa-48e6-9611-4b2fa94d84a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:33.029056 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dffcb3-b7aa-48e6-9611-4b2fa94d84a4 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:33.029532 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dffcb3-b7aa-48e6-9611-4b2fa94d84a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:33.030016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dffcb3-b7aa-48e6-9611-4b2fa94d84a4 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:33.030364 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df22f7dd-bc86-4c23-9a7f-4f430cf47b9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:33.030996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df22f7dd-bc86-4c23-9a7f-4f430cf47b9c None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:33.031291 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df22f7dd-bc86-4c23-9a7f-4f430cf47b9c None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:33.031543 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df22f7dd-bc86-4c23-9a7f-4f430cf47b9c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:33.031796 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15dffcb3-b7aa-48e6-9611-4b2fa94d84a4 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:33.032335 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15dffcb3-b7aa-48e6-9611-4b2fa94d84a4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:33.033987 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15dffcb3-b7aa-48e6-9611-4b2fa94d84a4 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:33.043132 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df22f7dd-bc86-4c23-9a7f-4f430cf47b9c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:33.048192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 129/259] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:06:32 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:33.050235 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-df22f7dd-bc86-4c23-9a7f-4f430cf47b9c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:06:33.050235 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:33.050235 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:33.050235 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:33.050235 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:33.050235 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:33.050235 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:33.050235 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:33.050235 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:33.050235 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:33.050235 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:33.050235 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:33.050235 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:33.050235 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:33.050235 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:33.050235 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:33.051647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:33.051647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:33.051647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:33.051647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:33.051647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:33.051647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:33.051647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:33.051647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:33.051647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:33.051647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:33.051647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:33.051647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:33.051647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:33.051647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:33.051647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:33.051647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:33.052476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:33.052476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:33.052476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:33.052476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:33.052476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:33.052476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:33.052476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:33.052476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:33.052476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:33.052476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:33.052476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:06:33.052476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:06:33.052476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 131/260] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:06:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:06:33.057621 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dffcb3-b7aa-48e6-9611-4b2fa94d84a4 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:33.059983 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df22f7dd-bc86-4c23-9a7f-4f430cf47b9c None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:33.159850 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:33.179497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df22f7dd-bc86-4c23-9a7f-4f430cf47b9c None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:33.197065 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dffcb3-b7aa-48e6-9611-4b2fa94d84a4 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:33.235686 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:33.252633 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcfc1170-cf44-43ee-9fa5-cdfda016d7e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:33.253247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcfc1170-cf44-43ee-9fa5-cdfda016d7e0 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:33.253525 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcfc1170-cf44-43ee-9fa5-cdfda016d7e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:33.253848 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcfc1170-cf44-43ee-9fa5-cdfda016d7e0 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:33.254526 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcfc1170-cf44-43ee-9fa5-cdfda016d7e0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:33.254830 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcfc1170-cf44-43ee-9fa5-cdfda016d7e0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:33.256808 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcfc1170-cf44-43ee-9fa5-cdfda016d7e0 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:33.266284 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 132/261] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:33.266757 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bf5d5d-fb9d-4682-8766-b0ef015c2dff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:33.267479 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bf5d5d-fb9d-4682-8766-b0ef015c2dff None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:33.267761 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bf5d5d-fb9d-4682-8766-b0ef015c2dff None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:33.268039 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bf5d5d-fb9d-4682-8766-b0ef015c2dff None admin] PATH_INFO: `/v3/users/1329a7ba9b3d48149f5a05ddeaf4bd31` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:33.270536 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7bf5d5d-fb9d-4682-8766-b0ef015c2dff None admin] RBAC: Authorizing `identity:get_user(user_id=1329a7ba9b3d48149f5a05ddeaf4bd31)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:33.272605 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7bf5d5d-fb9d-4682-8766-b0ef015c2dff None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:33.272732 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcfc1170-cf44-43ee-9fa5-cdfda016d7e0 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:33.274470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 130/262] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:33 2022] GET /identity/v3/users/1329a7ba9b3d48149f5a05ddeaf4bd31 => generated 283 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:33.279781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bf5d5d-fb9d-4682-8766-b0ef015c2dff None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:33.462197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcfc1170-cf44-43ee-9fa5-cdfda016d7e0 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:33.485720 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bf5d5d-fb9d-4682-8766-b0ef015c2dff None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:33.526888 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1957ad-f030-44cf-bfe2-9bd842d4c86f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:33.527542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1957ad-f030-44cf-bfe2-9bd842d4c86f None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:33.527827 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1957ad-f030-44cf-bfe2-9bd842d4c86f None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:33.528151 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1957ad-f030-44cf-bfe2-9bd842d4c86f None admin] PATH_INFO: `/v3/users/1329a7ba9b3d48149f5a05ddeaf4bd31` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:33.530496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1957ad-f030-44cf-bfe2-9bd842d4c86f None admin] RBAC: Authorizing `identity:get_user(user_id=1329a7ba9b3d48149f5a05ddeaf4bd31)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:33.532168 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1957ad-f030-44cf-bfe2-9bd842d4c86f None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:33.536565 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 133/263] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:33 2022] GET /identity/v3/users/1329a7ba9b3d48149f5a05ddeaf4bd31 => generated 283 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:33.545373 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1957ad-f030-44cf-bfe2-9bd842d4c86f None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:33.562189 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e67322b-e672-4f3c-b9c9-c0b6736cfac4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:33.563014 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e67322b-e672-4f3c-b9c9-c0b6736cfac4 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:33.563299 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e67322b-e672-4f3c-b9c9-c0b6736cfac4 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:33.563575 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e67322b-e672-4f3c-b9c9-c0b6736cfac4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:33.583594 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:33.657591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:33.705597 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e67322b-e672-4f3c-b9c9-c0b6736cfac4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:33.707356 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e67322b-e672-4f3c-b9c9-c0b6736cfac4 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:33.763780 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1957ad-f030-44cf-bfe2-9bd842d4c86f None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:33.829700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276e9be7-fbc6-428a-9e20-956a171284a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:33.830252 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276e9be7-fbc6-428a-9e20-956a171284a5 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:33.830534 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276e9be7-fbc6-428a-9e20-956a171284a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:33.830808 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276e9be7-fbc6-428a-9e20-956a171284a5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:33.917833 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 131/264] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 638 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 27 04:06:33.920178 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276e9be7-fbc6-428a-9e20-956a171284a5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:33.921699 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276e9be7-fbc6-428a-9e20-956a171284a5 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:33.924183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e67322b-e672-4f3c-b9c9-c0b6736cfac4 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:33.942148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:34.097795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:34.117417 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 134/265] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 572 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:06:34.124089 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276e9be7-fbc6-428a-9e20-956a171284a5 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:34.144525 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e67322b-e672-4f3c-b9c9-c0b6736cfac4 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:34.235583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc4aafe-23c7-4e38-bf84-a9565e91a6d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:34.236239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc4aafe-23c7-4e38-bf84-a9565e91a6d6 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:34.236511 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc4aafe-23c7-4e38-bf84-a9565e91a6d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:34.236784 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc4aafe-23c7-4e38-bf84-a9565e91a6d6 None admin] PATH_INFO: `/v3/projects/bbe6cda6a9564434b77be066e776bd55` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:34.242559 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc4aafe-23c7-4e38-bf84-a9565e91a6d6 None admin] RBAC: Authorizing `identity:get_project(project_id=bbe6cda6a9564434b77be066e776bd55)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:34.244051 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc4aafe-23c7-4e38-bf84-a9565e91a6d6 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:34.246084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 132/266] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:33 2022] GET /identity/v3/projects/bbe6cda6a9564434b77be066e776bd55 => generated 302 bytes in 322 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:34.253064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc4aafe-23c7-4e38-bf84-a9565e91a6d6 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:34.345462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276e9be7-fbc6-428a-9e20-956a171284a5 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:34.375105 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:34.413465 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e65eb73-aada-4258-9149-a84342f0737c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:34.414306 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e65eb73-aada-4258-9149-a84342f0737c None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:34.414585 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e65eb73-aada-4258-9149-a84342f0737c None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:34.414922 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e65eb73-aada-4258-9149-a84342f0737c None admin] PATH_INFO: `/v3/projects/bbe6cda6a9564434b77be066e776bd55` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:34.425498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e65eb73-aada-4258-9149-a84342f0737c None admin] RBAC: Authorizing `identity:get_project(project_id=bbe6cda6a9564434b77be066e776bd55)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:34.425498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e65eb73-aada-4258-9149-a84342f0737c None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:34.425498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 135/267] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:34 2022] GET /identity/v3/projects/bbe6cda6a9564434b77be066e776bd55 => generated 302 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:34.425834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77398ea-e7c3-46d9-8129-0fb8565e3b5e None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:34.426086 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77398ea-e7c3-46d9-8129-0fb8565e3b5e None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:34.426340 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77398ea-e7c3-46d9-8129-0fb8565e3b5e None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:34.427693 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 136/268] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:34.431292 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77398ea-e7c3-46d9-8129-0fb8565e3b5e None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:34.478991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc4aafe-23c7-4e38-bf84-a9565e91a6d6 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:34.541475 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185ae5de-9231-4ea7-ad1c-e1f0977e3c12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:34.542168 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185ae5de-9231-4ea7-ad1c-e1f0977e3c12 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:34.542454 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185ae5de-9231-4ea7-ad1c-e1f0977e3c12 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:34.542732 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185ae5de-9231-4ea7-ad1c-e1f0977e3c12 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:34.543458 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185ae5de-9231-4ea7-ad1c-e1f0977e3c12 None admin] RBAC: Adding query filter params (role.id=2d9f391485604d0b952b7c844c3a3266, scope.project.id=bbe6cda6a9564434b77be066e776bd55, user.id=1329a7ba9b3d48149f5a05ddeaf4bd31) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:34.543872 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185ae5de-9231-4ea7-ad1c-e1f0977e3c12 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:34.545432 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185ae5de-9231-4ea7-ad1c-e1f0977e3c12 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:34.552037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 133/269] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:06:34 2022] GET /identity/v3/role_assignments?user.id=1329a7ba9b3d48149f5a05ddeaf4bd31&scope.project.id=bbe6cda6a9564434b77be066e776bd55&role.id=2d9f391485604d0b952b7c844c3a3266 => generated 608 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:34.553427 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc1ec0f-2aa8-431d-8fcb-d842c3591bdf None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:34.553710 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc1ec0f-2aa8-431d-8fcb-d842c3591bdf None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:34.554016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc1ec0f-2aa8-431d-8fcb-d842c3591bdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:34.574727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-0dc1ec0f-2aa8-431d-8fcb-d842c3591bdf None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:34.576005 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-0dc1ec0f-2aa8-431d-8fcb-d842c3591bdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:34.588520 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:34.595321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77398ea-e7c3-46d9-8129-0fb8565e3b5e None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:34.671973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06181710-783f-4722-9824-7262c98671dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:34.673342 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06181710-783f-4722-9824-7262c98671dd None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:34.673626 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06181710-783f-4722-9824-7262c98671dd None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:34.675800 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06181710-783f-4722-9824-7262c98671dd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:34.676636 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06181710-783f-4722-9824-7262c98671dd None admin] RBAC: Adding query filter params (role.id=e6a31562a18649f0809202c36a8e18ef, scope.project.id=bbe6cda6a9564434b77be066e776bd55, user.id=1329a7ba9b3d48149f5a05ddeaf4bd31) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:34.676989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06181710-783f-4722-9824-7262c98671dd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:34.680327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06181710-783f-4722-9824-7262c98671dd None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:34.692311 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 134/270] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:34.694833 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 137/271] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:06:34 2022] GET /identity/v3/role_assignments?user.id=1329a7ba9b3d48149f5a05ddeaf4bd31&scope.project.id=bbe6cda6a9564434b77be066e776bd55&role.id=e6a31562a18649f0809202c36a8e18ef => generated 608 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:34.947936 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be07f950-a69f-43f2-8497-544c0c323672 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:34.948272 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be07f950-a69f-43f2-8497-544c0c323672 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:34.948438 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be07f950-a69f-43f2-8497-544c0c323672 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:34.950011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 135/272] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:34.955629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc97423-dfc7-4bd9-a25b-baa777e8715e None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:34.955921 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc97423-dfc7-4bd9-a25b-baa777e8715e None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:34.956197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc97423-dfc7-4bd9-a25b-baa777e8715e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:34.976511 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-3cc97423-dfc7-4bd9-a25b-baa777e8715e None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:34.977723 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-3cc97423-dfc7-4bd9-a25b-baa777e8715e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:35.029645 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:35.055355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 138/273] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:06:35.122787 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be07f950-a69f-43f2-8497-544c0c323672 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:35.254211 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be07f950-a69f-43f2-8497-544c0c323672 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:35.297310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8aa28a7-1c21-4974-a162-c335638c6c1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:35.298036 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8aa28a7-1c21-4974-a162-c335638c6c1d None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:35.298367 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8aa28a7-1c21-4974-a162-c335638c6c1d None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:35.298650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8aa28a7-1c21-4974-a162-c335638c6c1d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:35.299756 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8aa28a7-1c21-4974-a162-c335638c6c1d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:35.301386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8aa28a7-1c21-4974-a162-c335638c6c1d None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:35.303565 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 136/274] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:06:35 2022] GET /identity/v3/domains/default => generated 199 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:35.309219 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc97423-dfc7-4bd9-a25b-baa777e8715e None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:35.364927 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8aa28a7-1c21-4974-a162-c335638c6c1d None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:35.384309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:35.439565 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc97423-dfc7-4bd9-a25b-baa777e8715e None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:35.478399 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d191b47e-c892-4eef-81f0-6997632e0a35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:35.479025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d191b47e-c892-4eef-81f0-6997632e0a35 None admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:35.479306 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d191b47e-c892-4eef-81f0-6997632e0a35 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:35.480117 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d191b47e-c892-4eef-81f0-6997632e0a35 None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:35.480827 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d191b47e-c892-4eef-81f0-6997632e0a35 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:35.482511 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d191b47e-c892-4eef-81f0-6997632e0a35 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:35.493606 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8aa28a7-1c21-4974-a162-c335638c6c1d None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:35.508297 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 139/275] 10.222.0.26 () {60 vars in 1132 bytes} [Tue Sep 27 04:06:35 2022] POST /identity/v3/users => generated 293 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:06:35.515268 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef28dd3-addf-4325-bbab-f5ab60382742 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:35.515576 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef28dd3-addf-4325-bbab-f5ab60382742 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:35.515812 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef28dd3-addf-4325-bbab-f5ab60382742 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:35.517364 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 140/276] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:35.522976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d808af-2482-47cb-b07a-2693b1034671 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:35.523284 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d808af-2482-47cb-b07a-2693b1034671 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:35.523519 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d808af-2482-47cb-b07a-2693b1034671 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:35.544536 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-98d808af-2482-47cb-b07a-2693b1034671 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:35.545774 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-98d808af-2482-47cb-b07a-2693b1034671 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:35.546113 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96557b34-87aa-4864-9181-539b3fb342f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:35.547015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96557b34-87aa-4864-9181-539b3fb342f3 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:35.547521 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96557b34-87aa-4864-9181-539b3fb342f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:35.548472 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96557b34-87aa-4864-9181-539b3fb342f3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:35.568774 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96557b34-87aa-4864-9181-539b3fb342f3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:35.576758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-96557b34-87aa-4864-9181-539b3fb342f3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:35.576758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:35.576758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:35.576758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:35.576758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:35.576758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:35.576758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:35.576758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:35.576758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:35.576758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:35.576758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:35.576758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:35.576758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:35.576758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:35.576758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:35.576758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:35.577813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:35.577813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:35.577813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:35.577813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:35.577813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:35.577813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:35.577813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:35.577813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:35.577813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:35.577813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:35.577813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:35.577813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:35.577813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:35.577813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:35.577813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:35.577813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:35.578634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:35.578634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:35.578634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:35.578634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:35.578634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:35.578634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:35.578634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:35.578634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:35.578634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:35.578634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:35.578634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:35.578634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:06:35.579235 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 137/277] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:35 2022] GET /identity/v3/roles/member => generated 84 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:06:35.583978 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:35.584912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d779f127-8a52-442e-aa3a-29f1ca94b98c None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:35.585203 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d779f127-8a52-442e-aa3a-29f1ca94b98c None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:35.586353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d779f127-8a52-442e-aa3a-29f1ca94b98c None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:35.587879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 138/278] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:35 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:06:35.593593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d779f127-8a52-442e-aa3a-29f1ca94b98c None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:35.628401 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 141/279] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:06:35.629769 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5b769d-3495-4d8c-bbc7-1b62662605bb None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:35.630125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5b769d-3495-4d8c-bbc7-1b62662605bb None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:35.630403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5b769d-3495-4d8c-bbc7-1b62662605bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:35.650830 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-3c5b769d-3495-4d8c-bbc7-1b62662605bb None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:35.652050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-3c5b769d-3495-4d8c-bbc7-1b62662605bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:35.677132 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:35.715580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d779f127-8a52-442e-aa3a-29f1ca94b98c None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:35.742399 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 142/280] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:35.762747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856d4fda-eaff-4beb-84fc-c8c6d023e231 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:35.763289 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856d4fda-eaff-4beb-84fc-c8c6d023e231 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:35.763570 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856d4fda-eaff-4beb-84fc-c8c6d023e231 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:35.763892 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856d4fda-eaff-4beb-84fc-c8c6d023e231 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:35.764560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856d4fda-eaff-4beb-84fc-c8c6d023e231 None admin] RBAC: Adding query filter params (name=member) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:35.764909 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856d4fda-eaff-4beb-84fc-c8c6d023e231 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:35.766465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856d4fda-eaff-4beb-84fc-c8c6d023e231 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:35.775881 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 139/281] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:35 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:35.781465 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5b769d-3495-4d8c-bbc7-1b62662605bb None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:35.880012 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:35.886652 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5b769d-3495-4d8c-bbc7-1b62662605bb None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:35.910691 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856d4fda-eaff-4beb-84fc-c8c6d023e231 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:35.924364 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91116e96-861e-493c-ab81-9c2264cafafd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:35.925101 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91116e96-861e-493c-ab81-9c2264cafafd None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:35.925382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91116e96-861e-493c-ab81-9c2264cafafd None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:35.925651 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91116e96-861e-493c-ab81-9c2264cafafd None admin] PATH_INFO: `/v3/users/94f4bd0918674ce596e3c0cfa9d7eb2c` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:35.940166 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91116e96-861e-493c-ab81-9c2264cafafd None admin] RBAC: Authorizing `identity:get_user(user_id=94f4bd0918674ce596e3c0cfa9d7eb2c)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:35.941621 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91116e96-861e-493c-ab81-9c2264cafafd None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:35.944169 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 143/282] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:35 2022] GET /identity/v3/users/94f4bd0918674ce596e3c0cfa9d7eb2c => generated 293 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:35.950449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91116e96-861e-493c-ab81-9c2264cafafd None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:36.064877 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856d4fda-eaff-4beb-84fc-c8c6d023e231 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:36.079108 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91116e96-861e-493c-ab81-9c2264cafafd None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:36.101723 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6256304e-4b92-4045-8cdb-968c9b89d156 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:36.102384 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6256304e-4b92-4045-8cdb-968c9b89d156 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:36.102659 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6256304e-4b92-4045-8cdb-968c9b89d156 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:36.102978 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6256304e-4b92-4045-8cdb-968c9b89d156 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:36.104970 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:36.115437 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6256304e-4b92-4045-8cdb-968c9b89d156 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:36.117005 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fff5d1-c42c-4d14-8c6b-aa3abaabaff6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:36.117605 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fff5d1-c42c-4d14-8c6b-aa3abaabaff6 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:36.117907 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fff5d1-c42c-4d14-8c6b-aa3abaabaff6 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:36.118176 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fff5d1-c42c-4d14-8c6b-aa3abaabaff6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:36.118901 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97fff5d1-c42c-4d14-8c6b-aa3abaabaff6 None admin] RBAC: Adding query filter params (role.id=e3ce5a3697ec4292a446366d3caadeea, user.id=94f4bd0918674ce596e3c0cfa9d7eb2c, scope.system=all) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:36.119205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97fff5d1-c42c-4d14-8c6b-aa3abaabaff6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:36.120678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97fff5d1-c42c-4d14-8c6b-aa3abaabaff6 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:36.120944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-6256304e-4b92-4045-8cdb-968c9b89d156 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 04:06:36.120944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:36.120944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:36.120944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:36.120944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:36.120944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:36.120944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:36.120944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:36.120944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:36.120944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:36.120944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:36.120944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:36.120944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:36.120944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:36.120944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:36.120944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:36.121863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:36.121863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:36.121863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:36.121863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:36.121863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:36.121863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:36.121863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:36.121863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:36.121863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:36.121863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:36.121863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:36.121863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:36.121863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:36.121863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:36.121863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:36.121863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:36.123622 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:36.123622 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:36.123622 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:36.123622 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:36.123622 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:36.123622 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:36.123622 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:36.123622 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:36.123622 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:36.123622 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:36.123622 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 04:06:36.123622 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:06:36.123622 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 140/283] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:35 2022] GET /identity/v3/roles/reader => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:36.127823 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 144/284] 10.222.0.26 () {56 vars in 1308 bytes} [Tue Sep 27 04:06:35 2022] GET /identity/v3/role_assignments?user.id=94f4bd0918674ce596e3c0cfa9d7eb2c&scope.system=all&role.id=e3ce5a3697ec4292a446366d3caadeea => generated 226 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:36.128028 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6256304e-4b92-4045-8cdb-968c9b89d156 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:36.128728 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fff5d1-c42c-4d14-8c6b-aa3abaabaff6 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:36.154634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:36.321279 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fff5d1-c42c-4d14-8c6b-aa3abaabaff6 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:36.325487 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6256304e-4b92-4045-8cdb-968c9b89d156 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:36.383424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b4662f-69ec-4ab0-ba79-1311bbadb2cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:36.384404 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b4662f-69ec-4ab0-ba79-1311bbadb2cf None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:36.384882 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b4662f-69ec-4ab0-ba79-1311bbadb2cf None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:36.385911 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b4662f-69ec-4ab0-ba79-1311bbadb2cf None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:36.386788 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b4662f-69ec-4ab0-ba79-1311bbadb2cf None admin] RBAC: Adding query filter params (name=reader) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:36.387713 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b4662f-69ec-4ab0-ba79-1311bbadb2cf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:36.389576 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b4662f-69ec-4ab0-ba79-1311bbadb2cf None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:36.390920 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7601e51a-668e-4110-86e3-e5dcf346c368 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:36.391795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7601e51a-668e-4110-86e3-e5dcf346c368 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:36.393433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7601e51a-668e-4110-86e3-e5dcf346c368 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:36.393941 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7601e51a-668e-4110-86e3-e5dcf346c368 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:36.409290 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 145/285] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:36 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:36.412666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7601e51a-668e-4110-86e3-e5dcf346c368 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:36.413797 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b4662f-69ec-4ab0-ba79-1311bbadb2cf None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:36.430895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-7601e51a-668e-4110-86e3-e5dcf346c368 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:36.430895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:36.430895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:36.430895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:36.430895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:36.430895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:36.430895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:36.430895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:36.430895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:36.430895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:36.430895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:36.430895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:36.430895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:36.430895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:36.430895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:36.430895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:36.431820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:36.431820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:36.431820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:36.431820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:36.431820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:36.431820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:36.431820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:36.431820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:36.431820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:36.431820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:36.431820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:36.431820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:36.431820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:36.431820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:36.431820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:36.431820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:36.432589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:36.432589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:36.432589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:36.432589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:36.432589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:36.432589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:36.432589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:36.432589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:36.432589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:36.432589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:36.432589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:36.432589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:06:36.432589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 141/286] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:36 2022] GET /identity/v3/roles/member => generated 84 bytes in 309 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:36.439604 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7601e51a-668e-4110-86e3-e5dcf346c368 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:36.539298 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:36.554118 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:36.588878 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b4662f-69ec-4ab0-ba79-1311bbadb2cf None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:36.623677 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7601e51a-668e-4110-86e3-e5dcf346c368 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:36.652513 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366d27ea-1bb6-400b-816d-5abe85ddd162 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:36.653504 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366d27ea-1bb6-400b-816d-5abe85ddd162 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:36.653992 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366d27ea-1bb6-400b-816d-5abe85ddd162 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:36.655119 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366d27ea-1bb6-400b-816d-5abe85ddd162 None admin] PATH_INFO: `/v3/users/495345eb577b47a79c3fda23f8bc23eb` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:36.656640 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366d27ea-1bb6-400b-816d-5abe85ddd162 None admin] RBAC: Authorizing `identity:get_user(user_id=495345eb577b47a79c3fda23f8bc23eb)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:36.660807 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366d27ea-1bb6-400b-816d-5abe85ddd162 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:36.662987 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 146/287] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:36 2022] GET /identity/v3/users/495345eb577b47a79c3fda23f8bc23eb => generated 297 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:36.669328 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366d27ea-1bb6-400b-816d-5abe85ddd162 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:36.691264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4127a5-024e-4d1d-8023-8cee323542e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:36.692201 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4127a5-024e-4d1d-8023-8cee323542e7 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:36.693334 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4127a5-024e-4d1d-8023-8cee323542e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:36.693853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4127a5-024e-4d1d-8023-8cee323542e7 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:36.694752 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4127a5-024e-4d1d-8023-8cee323542e7 None admin] RBAC: Adding query filter params (name=member) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:36.695240 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4127a5-024e-4d1d-8023-8cee323542e7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:36.696874 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4127a5-024e-4d1d-8023-8cee323542e7 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:36.707533 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 142/288] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:36 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:36.713419 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4127a5-024e-4d1d-8023-8cee323542e7 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:36.846998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366d27ea-1bb6-400b-816d-5abe85ddd162 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:36.883164 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008879f8-25be-49cf-8b3e-798acfa1cfcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:36.884174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008879f8-25be-49cf-8b3e-798acfa1cfcb None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:36.884648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008879f8-25be-49cf-8b3e-798acfa1cfcb None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:36.885136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008879f8-25be-49cf-8b3e-798acfa1cfcb None admin] PATH_INFO: `/v3/projects/bbe6cda6a9564434b77be066e776bd55` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:36.886470 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008879f8-25be-49cf-8b3e-798acfa1cfcb None admin] RBAC: Authorizing `identity:get_project(project_id=bbe6cda6a9564434b77be066e776bd55)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:36.888160 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008879f8-25be-49cf-8b3e-798acfa1cfcb None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:36.890035 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 147/289] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:36 2022] GET /identity/v3/projects/bbe6cda6a9564434b77be066e776bd55 => generated 302 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:36.896959 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008879f8-25be-49cf-8b3e-798acfa1cfcb None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:36.906406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4127a5-024e-4d1d-8023-8cee323542e7 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:36.929613 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:36.939009 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:36.976463 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c687d0-4de5-4b88-b3c7-6ed45868d516 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:36.977098 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c687d0-4de5-4b88-b3c7-6ed45868d516 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:36.977373 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c687d0-4de5-4b88-b3c7-6ed45868d516 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:36.977643 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c687d0-4de5-4b88-b3c7-6ed45868d516 None admin] PATH_INFO: `/v3/users/4cc4e06ab0b441e785c85a4107e0dcae` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:36.979020 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c687d0-4de5-4b88-b3c7-6ed45868d516 None admin] RBAC: Authorizing `identity:get_user(user_id=4cc4e06ab0b441e785c85a4107e0dcae)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:36.980487 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c687d0-4de5-4b88-b3c7-6ed45868d516 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:36.984777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 143/290] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:36 2022] GET /identity/v3/users/4cc4e06ab0b441e785c85a4107e0dcae => generated 297 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:36.999229 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c687d0-4de5-4b88-b3c7-6ed45868d516 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:37.093489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008879f8-25be-49cf-8b3e-798acfa1cfcb None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:37.129932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3eb504-f5fb-4cfb-970a-ae70658d2607 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:37.130497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3eb504-f5fb-4cfb-970a-ae70658d2607 None admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:37.130851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3eb504-f5fb-4cfb-970a-ae70658d2607 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:37.131128 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3eb504-f5fb-4cfb-970a-ae70658d2607 None admin] PATH_INFO: `/v3/projects/bbe6cda6a9564434b77be066e776bd55/users/495345eb577b47a79c3fda23f8bc23eb/roles/9b729a0f423e47f7bc562c000780ef7b` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:37.133061 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3eb504-f5fb-4cfb-970a-ae70658d2607 None admin] RBAC: Authorizing `identity:create_grant(project_id=bbe6cda6a9564434b77be066e776bd55, user_id=495345eb577b47a79c3fda23f8bc23eb, role_id=9b729a0f423e47f7bc562c000780ef7b)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:37.134684 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3eb504-f5fb-4cfb-970a-ae70658d2607 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:37.162367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 148/291] 10.222.0.26 () {58 vars in 1439 bytes} [Tue Sep 27 04:06:36 2022] PUT /identity/v3/projects/bbe6cda6a9564434b77be066e776bd55/users/495345eb577b47a79c3fda23f8bc23eb/roles/9b729a0f423e47f7bc562c000780ef7b => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:06:37.198030 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c687d0-4de5-4b88-b3c7-6ed45868d516 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:37.237283 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7127c1-1213-4376-be31-bda9f9833bd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:37.237984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7127c1-1213-4376-be31-bda9f9833bd3 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:37.238262 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7127c1-1213-4376-be31-bda9f9833bd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:37.238587 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7127c1-1213-4376-be31-bda9f9833bd3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:37.298778 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:37.319073 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7127c1-1213-4376-be31-bda9f9833bd3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:37.320558 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7127c1-1213-4376-be31-bda9f9833bd3 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:37.450659 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 144/292] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 452 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:06:37.459524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3eb504-f5fb-4cfb-970a-ae70658d2607 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:37.571235 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:37.600740 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3eb504-f5fb-4cfb-970a-ae70658d2607 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:37.644953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245ec632-19bf-4374-b3ed-85e8c01948f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:37.645621 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245ec632-19bf-4374-b3ed-85e8c01948f3 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:37.645933 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245ec632-19bf-4374-b3ed-85e8c01948f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:37.646207 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245ec632-19bf-4374-b3ed-85e8c01948f3 None admin] PATH_INFO: `/v3/projects/bbe6cda6a9564434b77be066e776bd55` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:37.647325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245ec632-19bf-4374-b3ed-85e8c01948f3 None admin] RBAC: Authorizing `identity:get_project(project_id=bbe6cda6a9564434b77be066e776bd55)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:37.648763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245ec632-19bf-4374-b3ed-85e8c01948f3 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:37.650844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 149/293] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:37 2022] GET /identity/v3/projects/bbe6cda6a9564434b77be066e776bd55 => generated 302 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:37.657183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7127c1-1213-4376-be31-bda9f9833bd3 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:37.759898 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:37.783585 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7127c1-1213-4376-be31-bda9f9833bd3 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:37.829728 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd795b25-3668-44cc-a6c6-3c27c2f58547 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:37.830735 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd795b25-3668-44cc-a6c6-3c27c2f58547 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:37.831214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd795b25-3668-44cc-a6c6-3c27c2f58547 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:37.831682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd795b25-3668-44cc-a6c6-3c27c2f58547 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:37.832595 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd795b25-3668-44cc-a6c6-3c27c2f58547 None admin] RBAC: Adding query filter params (role.id=e3ce5a3697ec4292a446366d3caadeea, scope.project.id=bbe6cda6a9564434b77be066e776bd55, user.id=4cc4e06ab0b441e785c85a4107e0dcae) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:37.833644 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd795b25-3668-44cc-a6c6-3c27c2f58547 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:37.835346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd795b25-3668-44cc-a6c6-3c27c2f58547 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:37.846387 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 145/294] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:06:37 2022] GET /identity/v3/role_assignments?user.id=4cc4e06ab0b441e785c85a4107e0dcae&scope.project.id=bbe6cda6a9564434b77be066e776bd55&role.id=e3ce5a3697ec4292a446366d3caadeea => generated 608 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:38.194564 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e133ec4-bba5-4bef-841a-48237968e779 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:38.194944 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e133ec4-bba5-4bef-841a-48237968e779 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:38.195162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e133ec4-bba5-4bef-841a-48237968e779 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:38.196569 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 150/295] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:38.202053 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5861e48-38a5-448e-a223-96070f53575e None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:38.202491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5861e48-38a5-448e-a223-96070f53575e None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:38.202566 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5861e48-38a5-448e-a223-96070f53575e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:38.224772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-d5861e48-38a5-448e-a223-96070f53575e None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:38.225996 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d5861e48-38a5-448e-a223-96070f53575e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:38.299135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 146/296] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:38.305101 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8623b291-2fc7-4ba1-b87d-a8601f026e46 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:38.305101 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8623b291-2fc7-4ba1-b87d-a8601f026e46 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:38.305101 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8623b291-2fc7-4ba1-b87d-a8601f026e46 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:38.305679 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 151/297] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:38.311103 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d977de3-c357-4ecc-a3f2-b3a7f9ac3027 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:38.311615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d977de3-c357-4ecc-a3f2-b3a7f9ac3027 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:38.312088 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d977de3-c357-4ecc-a3f2-b3a7f9ac3027 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:38.334881 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-0d977de3-c357-4ecc-a3f2-b3a7f9ac3027 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:38.336388 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-0d977de3-c357-4ecc-a3f2-b3a7f9ac3027 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:38.439791 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 147/298] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:38.601404 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8623b291-2fc7-4ba1-b87d-a8601f026e46 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:38.608582 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c4f276-54b8-4a14-955b-0cac82c4c423 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:38.609242 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c4f276-54b8-4a14-955b-0cac82c4c423 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:38.609711 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c4f276-54b8-4a14-955b-0cac82c4c423 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:38.611662 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 148/299] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:06:38.618466 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e816d79d-ad7f-48f7-9fc9-b22b061f7f55 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:38.619069 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e816d79d-ad7f-48f7-9fc9-b22b061f7f55 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:38.619564 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e816d79d-ad7f-48f7-9fc9-b22b061f7f55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:38.640353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-e816d79d-ad7f-48f7-9fc9-b22b061f7f55 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:38.642393 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-e816d79d-ad7f-48f7-9fc9-b22b061f7f55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:38.644760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:38.719488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8623b291-2fc7-4ba1-b87d-a8601f026e46 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:38.732156 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 149/300] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:06:38.733896 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e816d79d-ad7f-48f7-9fc9-b22b061f7f55 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:38.768352 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ecccb8-2865-4a15-89e1-401f3f03a30a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:38.769295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ecccb8-2865-4a15-89e1-401f3f03a30a None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:38.769802 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ecccb8-2865-4a15-89e1-401f3f03a30a None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:38.770569 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ecccb8-2865-4a15-89e1-401f3f03a30a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:38.772716 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ecccb8-2865-4a15-89e1-401f3f03a30a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:38.774392 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ecccb8-2865-4a15-89e1-401f3f03a30a None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:38.776443 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 152/301] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:06:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:38.782536 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ecccb8-2865-4a15-89e1-401f3f03a30a None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:38.803931 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:38.854638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e816d79d-ad7f-48f7-9fc9-b22b061f7f55 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:38.893213 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa6d013-ae6f-4dea-953b-b46943dd7978 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:38.894117 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa6d013-ae6f-4dea-953b-b46943dd7978 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:38.894535 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa6d013-ae6f-4dea-953b-b46943dd7978 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:38.894971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa6d013-ae6f-4dea-953b-b46943dd7978 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:38.902714 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:38.907562 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa6d013-ae6f-4dea-953b-b46943dd7978 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:38.910572 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ecccb8-2865-4a15-89e1-401f3f03a30a None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:38.913997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-baa6d013-ae6f-4dea-953b-b46943dd7978 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 04:06:38.913997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:38.913997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:38.913997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:38.913997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:38.913997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:38.913997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:38.913997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:38.913997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:38.913997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:38.913997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:38.913997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:38.913997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:38.913997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:38.913997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:38.913997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:38.914876 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:38.914876 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:38.914876 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:38.914876 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:38.914876 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:38.914876 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:38.914876 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:38.914876 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:38.914876 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:38.914876 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:38.914876 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:38.914876 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:38.914876 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:38.914876 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:38.914876 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:38.914876 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:38.915646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:38.915646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:38.915646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:38.915646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:38.915646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:38.915646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:38.915646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:38.915646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:38.915646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:38.915646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:38.915646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 04:06:38.915646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:06:38.921923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 150/302] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:38 2022] GET /identity/v3/roles/reader => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:38.924139 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa6d013-ae6f-4dea-953b-b46943dd7978 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:38.950946 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b31eb3e-0863-4fba-b801-d58e1d630dd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:38.951809 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b31eb3e-0863-4fba-b801-d58e1d630dd6 None admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:38.952251 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b31eb3e-0863-4fba-b801-d58e1d630dd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:38.952758 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b31eb3e-0863-4fba-b801-d58e1d630dd6 None admin] PATH_INFO: `/v3/groups` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:38.953535 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b31eb3e-0863-4fba-b801-d58e1d630dd6 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:38.955190 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b31eb3e-0863-4fba-b801-d58e1d630dd6 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:38.975798 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 153/303] 10.222.0.26 () {60 vars in 1134 bytes} [Tue Sep 27 04:06:38 2022] POST /identity/v3/groups => generated 229 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:06:38.997107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b31eb3e-0863-4fba-b801-d58e1d630dd6 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:39.065440 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa6d013-ae6f-4dea-953b-b46943dd7978 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:39.080990 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:39.117924 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0808b715-a6c9-4cfa-ab51-307bfaafbfc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:39.119481 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0808b715-a6c9-4cfa-ab51-307bfaafbfc1 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:39.119756 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0808b715-a6c9-4cfa-ab51-307bfaafbfc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:39.120027 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0808b715-a6c9-4cfa-ab51-307bfaafbfc1 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:39.120704 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0808b715-a6c9-4cfa-ab51-307bfaafbfc1 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:39.121001 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0808b715-a6c9-4cfa-ab51-307bfaafbfc1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:39.122446 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0808b715-a6c9-4cfa-ab51-307bfaafbfc1 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:39.124381 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b31eb3e-0863-4fba-b801-d58e1d630dd6 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:39.132639 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 151/304] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:38 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:39.143554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0808b715-a6c9-4cfa-ab51-307bfaafbfc1 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:39.174788 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332a8aba-7736-4df0-8230-058ef0779fed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:39.175697 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332a8aba-7736-4df0-8230-058ef0779fed None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:39.176617 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332a8aba-7736-4df0-8230-058ef0779fed None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:39.177472 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332a8aba-7736-4df0-8230-058ef0779fed None admin] PATH_INFO: `/v3/roles/member` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:39.197904 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332a8aba-7736-4df0-8230-058ef0779fed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:39.204863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-332a8aba-7736-4df0-8230-058ef0779fed None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:39.204863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:39.204863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:39.204863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:39.204863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:39.204863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:39.204863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:39.204863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:39.204863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:39.204863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:39.204863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:39.204863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:39.204863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:39.204863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:39.204863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:39.204863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:39.206107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:39.206107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:39.206107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:39.206107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:39.206107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:39.206107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:39.206107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:39.206107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:39.206107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:39.206107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:39.206107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:39.206107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:39.206107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:39.206107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:39.206107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:39.206107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:39.206890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:39.206890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:39.206890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:39.206890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:39.206890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:39.206890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:39.206890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:39.206890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:39.206890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:39.206890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:39.206890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:39.206890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:06:39.207482 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 154/305] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:38 2022] GET /identity/v3/roles/member => generated 84 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:39.213878 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332a8aba-7736-4df0-8230-058ef0779fed None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:39.226322 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:39.270964 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0808b715-a6c9-4cfa-ab51-307bfaafbfc1 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:39.309278 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48922ea5-a36c-4b12-a0ca-531b32ccb6e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:39.310180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48922ea5-a36c-4b12-a0ca-531b32ccb6e3 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:39.310703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48922ea5-a36c-4b12-a0ca-531b32ccb6e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:39.311147 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48922ea5-a36c-4b12-a0ca-531b32ccb6e3 None admin] PATH_INFO: `/v3/users/16a4e626417f43e6a60ac07c6e263212` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:39.327329 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48922ea5-a36c-4b12-a0ca-531b32ccb6e3 None admin] RBAC: Authorizing `identity:get_user(user_id=16a4e626417f43e6a60ac07c6e263212)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:39.329191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48922ea5-a36c-4b12-a0ca-531b32ccb6e3 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:39.332114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 152/306] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:39 2022] GET /identity/v3/users/16a4e626417f43e6a60ac07c6e263212 => generated 293 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:39.338709 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48922ea5-a36c-4b12-a0ca-531b32ccb6e3 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:39.348170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332a8aba-7736-4df0-8230-058ef0779fed None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:39.388513 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:39.395079 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c56194-cedf-4d96-a728-761b9fb88a92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:39.396126 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c56194-cedf-4d96-a728-761b9fb88a92 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:39.396418 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c56194-cedf-4d96-a728-761b9fb88a92 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:39.396673 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c56194-cedf-4d96-a728-761b9fb88a92 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:39.397388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c56194-cedf-4d96-a728-761b9fb88a92 None admin] RBAC: Adding query filter params (name=member) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:39.397690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c56194-cedf-4d96-a728-761b9fb88a92 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:39.399196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c56194-cedf-4d96-a728-761b9fb88a92 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:39.414110 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 155/307] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:39 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:39.416471 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c56194-cedf-4d96-a728-761b9fb88a92 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:39.471561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48922ea5-a36c-4b12-a0ca-531b32ccb6e3 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:39.490604 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:39.509997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79bfd0ad-0744-4aa0-be6e-101d493bc669 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:39.510917 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bfd0ad-0744-4aa0-be6e-101d493bc669 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:39.511146 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bfd0ad-0744-4aa0-be6e-101d493bc669 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:39.511534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bfd0ad-0744-4aa0-be6e-101d493bc669 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:39.512349 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79bfd0ad-0744-4aa0-be6e-101d493bc669 None admin] RBAC: Adding query filter params (role.id=9b729a0f423e47f7bc562c000780ef7b, user.id=16a4e626417f43e6a60ac07c6e263212, scope.system=all) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:39.512748 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79bfd0ad-0744-4aa0-be6e-101d493bc669 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:39.514294 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79bfd0ad-0744-4aa0-be6e-101d493bc669 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:39.520014 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 153/308] 10.222.0.26 () {56 vars in 1308 bytes} [Tue Sep 27 04:06:39 2022] GET /identity/v3/role_assignments?user.id=16a4e626417f43e6a60ac07c6e263212&scope.system=all&role.id=9b729a0f423e47f7bc562c000780ef7b => generated 226 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:39.541261 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c56194-cedf-4d96-a728-761b9fb88a92 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:39.580785 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2c92d7-6e3f-43f1-b602-86e6519a8c61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:39.581694 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2c92d7-6e3f-43f1-b602-86e6519a8c61 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:39.582197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2c92d7-6e3f-43f1-b602-86e6519a8c61 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:39.582624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2c92d7-6e3f-43f1-b602-86e6519a8c61 None admin] PATH_INFO: `/v3/users/94f4bd0918674ce596e3c0cfa9d7eb2c` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:39.584615 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2c92d7-6e3f-43f1-b602-86e6519a8c61 None admin] RBAC: Authorizing `identity:get_user(user_id=94f4bd0918674ce596e3c0cfa9d7eb2c)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:39.586400 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2c92d7-6e3f-43f1-b602-86e6519a8c61 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:39.593082 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 156/309] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:39 2022] GET /identity/v3/users/94f4bd0918674ce596e3c0cfa9d7eb2c => generated 293 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:39.594703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79bfd0ad-0744-4aa0-be6e-101d493bc669 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:39.711633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79bfd0ad-0744-4aa0-be6e-101d493bc669 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:39.745835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa3676f-0f05-4af9-995f-ea29b5c8f217 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:39.746786 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa3676f-0f05-4af9-995f-ea29b5c8f217 None admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:39.747239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa3676f-0f05-4af9-995f-ea29b5c8f217 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:39.747768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa3676f-0f05-4af9-995f-ea29b5c8f217 None admin] PATH_INFO: `/v3/system/users/94f4bd0918674ce596e3c0cfa9d7eb2c/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:39.749393 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa3676f-0f05-4af9-995f-ea29b5c8f217 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=94f4bd0918674ce596e3c0cfa9d7eb2c, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:39.750980 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa3676f-0f05-4af9-995f-ea29b5c8f217 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:39.762284 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 154/310] 10.222.0.26 () {58 vars in 1334 bytes} [Tue Sep 27 04:06:39 2022] PUT /identity/v3/system/users/94f4bd0918674ce596e3c0cfa9d7eb2c/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:06:39.945880 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fece061-ad0b-4048-a44d-5081e971d2fc None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:39.946889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fece061-ad0b-4048-a44d-5081e971d2fc None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:39.947369 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fece061-ad0b-4048-a44d-5081e971d2fc None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:39.949172 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 157/311] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:39.954205 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8706e807-adf6-4553-9eaa-f48706d886ed None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:39.954497 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8706e807-adf6-4553-9eaa-f48706d886ed None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:39.954776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8706e807-adf6-4553-9eaa-f48706d886ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:39.976283 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-8706e807-adf6-4553-9eaa-f48706d886ed None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:39.977425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-8706e807-adf6-4553-9eaa-f48706d886ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:40.021461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:40.048438 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 155/312] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:40.324710 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fece061-ad0b-4048-a44d-5081e971d2fc None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:40.389220 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:40.436352 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fece061-ad0b-4048-a44d-5081e971d2fc None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:40.473373 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c224486f-4f27-46a0-9c58-e0d327e53532 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:40.474564 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c224486f-4f27-46a0-9c58-e0d327e53532 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:40.474905 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c224486f-4f27-46a0-9c58-e0d327e53532 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:40.475169 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c224486f-4f27-46a0-9c58-e0d327e53532 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:40.485644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c224486f-4f27-46a0-9c58-e0d327e53532 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:40.491489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-c224486f-4f27-46a0-9c58-e0d327e53532 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 04:06:40.491489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:40.491489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:40.491489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:40.491489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:40.491489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:40.491489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:40.491489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:40.491489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:40.491489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:40.491489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:40.491489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:40.491489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:40.491489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:40.491489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:40.491489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:40.492365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:40.492365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:40.492365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:40.492365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:40.492365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:40.492365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:40.492365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:40.492365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:40.492365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:40.492365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:40.492365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:40.492365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:40.492365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:40.492365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:40.492365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:40.492365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:40.493149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:40.493149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:40.493149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:40.493149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:40.493149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:40.493149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:40.493149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:40.493149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:40.493149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:40.493149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:40.493149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 04:06:40.493149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:06:40.493149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 158/313] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:40 2022] GET /identity/v3/roles/reader => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:40.500132 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8706e807-adf6-4553-9eaa-f48706d886ed None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:40.603070 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8706e807-adf6-4553-9eaa-f48706d886ed None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:40.641467 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee8c67d-e33e-46a9-8bfd-c5118b4eb2d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:40.642098 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee8c67d-e33e-46a9-8bfd-c5118b4eb2d6 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:40.642369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee8c67d-e33e-46a9-8bfd-c5118b4eb2d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:40.642635 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee8c67d-e33e-46a9-8bfd-c5118b4eb2d6 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:40.643289 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee8c67d-e33e-46a9-8bfd-c5118b4eb2d6 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:40.643584 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee8c67d-e33e-46a9-8bfd-c5118b4eb2d6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:40.645079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee8c67d-e33e-46a9-8bfd-c5118b4eb2d6 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:40.653779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 156/314] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:40 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:40.663039 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c224486f-4f27-46a0-9c58-e0d327e53532 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:40.768229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c224486f-4f27-46a0-9c58-e0d327e53532 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:40.806474 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:40.811324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0504a817-4d36-4960-b578-8597de06d0bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:40.812051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0504a817-4d36-4960-b578-8597de06d0bb None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:40.812505 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0504a817-4d36-4960-b578-8597de06d0bb None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:40.812882 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0504a817-4d36-4960-b578-8597de06d0bb None admin] PATH_INFO: `/v3/users/495345eb577b47a79c3fda23f8bc23eb` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:40.814548 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0504a817-4d36-4960-b578-8597de06d0bb None admin] RBAC: Authorizing `identity:get_user(user_id=495345eb577b47a79c3fda23f8bc23eb)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:40.816797 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0504a817-4d36-4960-b578-8597de06d0bb None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:40.822065 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 159/315] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:40 2022] GET /identity/v3/users/495345eb577b47a79c3fda23f8bc23eb => generated 297 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:40.826222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee8c67d-e33e-46a9-8bfd-c5118b4eb2d6 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:40.871160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:40.932451 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee8c67d-e33e-46a9-8bfd-c5118b4eb2d6 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:40.965803 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62afcaa9-7132-44b5-845a-b8cff17ab971 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:40.966409 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62afcaa9-7132-44b5-845a-b8cff17ab971 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:40.966686 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62afcaa9-7132-44b5-845a-b8cff17ab971 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:40.966955 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62afcaa9-7132-44b5-845a-b8cff17ab971 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:41.039775 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62afcaa9-7132-44b5-845a-b8cff17ab971 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:41.041216 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62afcaa9-7132-44b5-845a-b8cff17ab971 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:41.080037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:41.143544 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 157/316] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:06:41.150244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0504a817-4d36-4960-b578-8597de06d0bb None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:41.256838 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0504a817-4d36-4960-b578-8597de06d0bb None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:41.277519 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f0ceb3-2e34-4395-913a-dadc7bf5235b None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:41.277831 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f0ceb3-2e34-4395-913a-dadc7bf5235b None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:41.278092 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f0ceb3-2e34-4395-913a-dadc7bf5235b None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:41.279559 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 158/317] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:06:41.284658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1313b401-9fac-4537-b443-d94f8da20ab8 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:41.284920 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1313b401-9fac-4537-b443-d94f8da20ab8 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:41.285190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1313b401-9fac-4537-b443-d94f8da20ab8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:41.297006 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8378ca48-2174-4593-8aed-73f97a7152ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:41.297977 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8378ca48-2174-4593-8aed-73f97a7152ae None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:41.298460 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8378ca48-2174-4593-8aed-73f97a7152ae None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:41.298929 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8378ca48-2174-4593-8aed-73f97a7152ae None admin] PATH_INFO: `/v3/projects/bbe6cda6a9564434b77be066e776bd55` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:41.300260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8378ca48-2174-4593-8aed-73f97a7152ae None admin] RBAC: Authorizing `identity:get_project(project_id=bbe6cda6a9564434b77be066e776bd55)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:41.301931 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8378ca48-2174-4593-8aed-73f97a7152ae None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:41.303459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-1313b401-9fac-4537-b443-d94f8da20ab8 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:41.304124 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 160/318] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:41 2022] GET /identity/v3/projects/bbe6cda6a9564434b77be066e776bd55 => generated 302 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:41.306470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-1313b401-9fac-4537-b443-d94f8da20ab8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:41.310154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8378ca48-2174-4593-8aed-73f97a7152ae None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:41.380477 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:41.401240 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 159/319] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:41.424183 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8378ca48-2174-4593-8aed-73f97a7152ae None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:41.459394 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066bdcbb-b70d-4ebf-8be3-6ab0082d3728 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:41.460000 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066bdcbb-b70d-4ebf-8be3-6ab0082d3728 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:41.460280 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066bdcbb-b70d-4ebf-8be3-6ab0082d3728 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:41.460556 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066bdcbb-b70d-4ebf-8be3-6ab0082d3728 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:41.461287 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-066bdcbb-b70d-4ebf-8be3-6ab0082d3728 None admin] RBAC: Adding query filter params (role.id=9b729a0f423e47f7bc562c000780ef7b, scope.project.id=bbe6cda6a9564434b77be066e776bd55, user.id=495345eb577b47a79c3fda23f8bc23eb) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:41.461594 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-066bdcbb-b70d-4ebf-8be3-6ab0082d3728 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:41.463055 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-066bdcbb-b70d-4ebf-8be3-6ab0082d3728 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:41.469163 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 161/320] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:06:41 2022] GET /identity/v3/role_assignments?user.id=495345eb577b47a79c3fda23f8bc23eb&scope.project.id=bbe6cda6a9564434b77be066e776bd55&role.id=9b729a0f423e47f7bc562c000780ef7b => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:41.678501 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1313b401-9fac-4537-b443-d94f8da20ab8 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:41.759872 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:41.794343 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1313b401-9fac-4537-b443-d94f8da20ab8 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:41.816514 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782cd1f0-3711-44d2-a745-b182eea9f70e None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:41.816794 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782cd1f0-3711-44d2-a745-b182eea9f70e None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:41.817255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782cd1f0-3711-44d2-a745-b182eea9f70e None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:41.819444 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 162/321] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:06:41.824250 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b59dd6-2850-4ece-97d4-1f340d896c9e None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:41.824672 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b59dd6-2850-4ece-97d4-1f340d896c9e None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:41.825082 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b59dd6-2850-4ece-97d4-1f340d896c9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:41.830687 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46cedda4-fe85-4148-a48f-59cd05161747 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:41.831333 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cedda4-fe85-4148-a48f-59cd05161747 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:41.831774 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cedda4-fe85-4148-a48f-59cd05161747 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:41.832056 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cedda4-fe85-4148-a48f-59cd05161747 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:41.834058 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46cedda4-fe85-4148-a48f-59cd05161747 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:41.835492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46cedda4-fe85-4148-a48f-59cd05161747 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:41.837189 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 160/322] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:06:41 2022] GET /identity/v3/domains/default => generated 199 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:41.842398 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46cedda4-fe85-4148-a48f-59cd05161747 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:41.846633 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-d2b59dd6-2850-4ece-97d4-1f340d896c9e None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:41.847810 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-d2b59dd6-2850-4ece-97d4-1f340d896c9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:41.918052 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 163/323] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:06:41.943359 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46cedda4-fe85-4148-a48f-59cd05161747 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:41.981087 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8764b5c1-42f6-4661-8769-eb04e2fdd566 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:41.981692 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8764b5c1-42f6-4661-8769-eb04e2fdd566 None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:41.982002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8764b5c1-42f6-4661-8769-eb04e2fdd566 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:41.982265 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8764b5c1-42f6-4661-8769-eb04e2fdd566 None admin] PATH_INFO: `/v3/groups` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:41.982893 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8764b5c1-42f6-4661-8769-eb04e2fdd566 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:41.984312 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8764b5c1-42f6-4661-8769-eb04e2fdd566 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:41.990779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:42.013770 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 161/324] 10.222.0.26 () {60 vars in 1134 bytes} [Tue Sep 27 04:06:41 2022] POST /identity/v3/groups => generated 226 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:06:42.061007 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e55fa3-fa69-4ee4-9834-f6f09c0021ca None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:42.061310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e55fa3-fa69-4ee4-9834-f6f09c0021ca None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:42.061621 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e55fa3-fa69-4ee4-9834-f6f09c0021ca None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:42.063265 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 164/325] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:42.069013 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23085927-0e7a-4398-85ec-8de885fb63c3 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:42.069320 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23085927-0e7a-4398-85ec-8de885fb63c3 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:42.069567 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23085927-0e7a-4398-85ec-8de885fb63c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:42.088594 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-23085927-0e7a-4398-85ec-8de885fb63c3 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:42.089901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-23085927-0e7a-4398-85ec-8de885fb63c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:42.165789 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 162/326] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:06:42.187134 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e55fa3-fa69-4ee4-9834-f6f09c0021ca None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:42.241507 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:42.296929 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e55fa3-fa69-4ee4-9834-f6f09c0021ca None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:42.334691 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5306bb6-6f0b-46d3-b172-ab9c17a5dc2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:42.335327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5306bb6-6f0b-46d3-b172-ab9c17a5dc2b None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:42.335605 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5306bb6-6f0b-46d3-b172-ab9c17a5dc2b None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:42.335878 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5306bb6-6f0b-46d3-b172-ab9c17a5dc2b None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:42.348072 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5306bb6-6f0b-46d3-b172-ab9c17a5dc2b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:42.353604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-d5306bb6-6f0b-46d3-b172-ab9c17a5dc2b None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 04:06:42.353604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:42.353604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:42.353604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:42.353604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:42.353604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:42.353604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:42.353604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:42.353604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:42.353604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:42.353604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:42.353604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:42.353604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:42.353604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:42.353604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:42.353604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:42.354498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:42.354498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:42.354498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:42.354498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:42.354498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:42.354498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:42.354498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:42.354498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:42.354498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:42.354498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:42.354498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:42.354498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:42.354498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:42.354498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:42.354498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:42.354498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:42.355275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:42.355275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:42.355275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:42.355275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:42.355275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:42.355275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:42.355275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:42.355275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:42.355275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:42.355275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:42.355275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 04:06:42.355275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:06:42.355275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 165/327] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:42 2022] GET /identity/v3/roles/reader => generated 84 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:42.361914 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23085927-0e7a-4398-85ec-8de885fb63c3 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:42.436538 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5306bb6-6f0b-46d3-b172-ab9c17a5dc2b None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:42.471562 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23085927-0e7a-4398-85ec-8de885fb63c3 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:42.501307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:42.513780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f369a982-873a-44fa-9641-08f6b213369d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:42.514739 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f369a982-873a-44fa-9641-08f6b213369d None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:42.515155 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f369a982-873a-44fa-9641-08f6b213369d None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:42.515564 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f369a982-873a-44fa-9641-08f6b213369d None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:42.518421 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f369a982-873a-44fa-9641-08f6b213369d None admin] RBAC: Adding query filter params (name=reader) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:42.518421 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f369a982-873a-44fa-9641-08f6b213369d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:42.519530 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f369a982-873a-44fa-9641-08f6b213369d None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:42.530935 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 163/328] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:42 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:42.537706 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f369a982-873a-44fa-9641-08f6b213369d None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:42.580182 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5306bb6-6f0b-46d3-b172-ab9c17a5dc2b None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:42.598013 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:42.635921 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf3a567-b5b0-429b-bd06-1e2e93ee03ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:42.636561 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf3a567-b5b0-429b-bd06-1e2e93ee03ac None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:42.636838 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf3a567-b5b0-429b-bd06-1e2e93ee03ac None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:42.637107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf3a567-b5b0-429b-bd06-1e2e93ee03ac None admin] PATH_INFO: `/v3/roles/member` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:42.650444 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf3a567-b5b0-429b-bd06-1e2e93ee03ac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:42.657253 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-6bf3a567-b5b0-429b-bd06-1e2e93ee03ac None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:42.657253 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:42.657253 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:42.657253 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:42.657253 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:42.657253 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:42.657253 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:42.657253 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:42.657253 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:42.657253 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:42.657253 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:42.657253 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:42.657253 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:42.657253 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:42.657253 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:42.657253 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:42.658367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:42.658367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:42.658367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:42.658367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:42.658367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:42.658367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:42.658367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:42.658367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:42.658367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:42.658367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:42.658367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:42.658367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:42.658367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:42.658367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:42.658367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:42.658367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:42.659162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:42.659162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:42.659162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:42.659162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:42.659162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:42.659162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:42.659162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:42.659162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:42.659162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:42.659162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:42.659162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:42.659162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:06:42.659162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 166/329] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:42 2022] GET /identity/v3/roles/member => generated 84 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:06:42.664524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf3a567-b5b0-429b-bd06-1e2e93ee03ac None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:42.683783 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f369a982-873a-44fa-9641-08f6b213369d None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:42.724528 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8469b7d-0aae-400c-a29f-7538d8efbf35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:42.725499 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8469b7d-0aae-400c-a29f-7538d8efbf35 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:42.725979 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8469b7d-0aae-400c-a29f-7538d8efbf35 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:42.726432 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8469b7d-0aae-400c-a29f-7538d8efbf35 None admin] PATH_INFO: `/v3/users/16a4e626417f43e6a60ac07c6e263212` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:42.728042 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8469b7d-0aae-400c-a29f-7538d8efbf35 None admin] RBAC: Authorizing `identity:get_user(user_id=16a4e626417f43e6a60ac07c6e263212)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:42.730175 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8469b7d-0aae-400c-a29f-7538d8efbf35 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:42.732365 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 164/330] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:42 2022] GET /identity/v3/users/16a4e626417f43e6a60ac07c6e263212 => generated 293 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:42.739323 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8469b7d-0aae-400c-a29f-7538d8efbf35 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:42.798821 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf3a567-b5b0-429b-bd06-1e2e93ee03ac None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:42.808980 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:42.854011 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641f54fa-1304-438f-a958-50a4ffb22d31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:42.854626 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641f54fa-1304-438f-a958-50a4ffb22d31 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:42.854905 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641f54fa-1304-438f-a958-50a4ffb22d31 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:42.855176 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641f54fa-1304-438f-a958-50a4ffb22d31 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:42.855850 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641f54fa-1304-438f-a958-50a4ffb22d31 None admin] RBAC: Adding query filter params (name=member) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:42.856151 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641f54fa-1304-438f-a958-50a4ffb22d31 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:42.857538 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641f54fa-1304-438f-a958-50a4ffb22d31 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:42.870153 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 167/331] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:42 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:42.870971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8469b7d-0aae-400c-a29f-7538d8efbf35 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:42.876210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641f54fa-1304-438f-a958-50a4ffb22d31 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:42.916285 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a798b693-61b7-4306-873e-c6d22bd95f67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:42.918010 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a798b693-61b7-4306-873e-c6d22bd95f67 None admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:42.918765 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a798b693-61b7-4306-873e-c6d22bd95f67 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:42.919237 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a798b693-61b7-4306-873e-c6d22bd95f67 None admin] PATH_INFO: `/v3/system/users/16a4e626417f43e6a60ac07c6e263212/roles/9b729a0f423e47f7bc562c000780ef7b` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:42.921524 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a798b693-61b7-4306-873e-c6d22bd95f67 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=16a4e626417f43e6a60ac07c6e263212, role_id=9b729a0f423e47f7bc562c000780ef7b)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:42.923173 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a798b693-61b7-4306-873e-c6d22bd95f67 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:42.930112 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:42.935185 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 165/332] 10.222.0.26 () {58 vars in 1334 bytes} [Tue Sep 27 04:06:42 2022] PUT /identity/v3/system/users/16a4e626417f43e6a60ac07c6e263212/roles/9b729a0f423e47f7bc562c000780ef7b => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:06:43.016988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641f54fa-1304-438f-a958-50a4ffb22d31 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:43.063109 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b153aca1-7815-463c-afd7-b58b5104f2a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:43.064072 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b153aca1-7815-463c-afd7-b58b5104f2a3 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:43.064576 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b153aca1-7815-463c-afd7-b58b5104f2a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:43.065042 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b153aca1-7815-463c-afd7-b58b5104f2a3 None admin] PATH_INFO: `/v3/users/94f4bd0918674ce596e3c0cfa9d7eb2c` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:43.066503 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b153aca1-7815-463c-afd7-b58b5104f2a3 None admin] RBAC: Authorizing `identity:get_user(user_id=94f4bd0918674ce596e3c0cfa9d7eb2c)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:43.068809 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b153aca1-7815-463c-afd7-b58b5104f2a3 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:43.070821 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 168/333] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:42 2022] GET /identity/v3/users/94f4bd0918674ce596e3c0cfa9d7eb2c => generated 293 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:43.076746 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a798b693-61b7-4306-873e-c6d22bd95f67 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:43.196648 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a798b693-61b7-4306-873e-c6d22bd95f67 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:43.219311 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:43.240470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d886eec2-8e1c-44ab-b4f3-1e29ec1c14af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:43.241077 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d886eec2-8e1c-44ab-b4f3-1e29ec1c14af None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:43.241353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d886eec2-8e1c-44ab-b4f3-1e29ec1c14af None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:43.241621 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d886eec2-8e1c-44ab-b4f3-1e29ec1c14af None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:43.242377 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d886eec2-8e1c-44ab-b4f3-1e29ec1c14af None admin] RBAC: Adding query filter params (role.id=e3ce5a3697ec4292a446366d3caadeea, user.id=94f4bd0918674ce596e3c0cfa9d7eb2c, scope.system=all) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:43.242680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d886eec2-8e1c-44ab-b4f3-1e29ec1c14af None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:43.244118 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d886eec2-8e1c-44ab-b4f3-1e29ec1c14af None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:43.263365 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 166/334] 10.222.0.26 () {56 vars in 1308 bytes} [Tue Sep 27 04:06:43 2022] GET /identity/v3/role_assignments?user.id=94f4bd0918674ce596e3c0cfa9d7eb2c&scope.system=all&role.id=e3ce5a3697ec4292a446366d3caadeea => generated 510 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:44.602959 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f405c17-93b9-4c0a-b62e-cfb55588d004 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:44.603796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f405c17-93b9-4c0a-b62e-cfb55588d004 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:44.604252 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f405c17-93b9-4c0a-b62e-cfb55588d004 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:44.605725 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 169/335] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:44.611084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496330e5-3f40-44d7-974d-dbc3d80fe740 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:44.611491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496330e5-3f40-44d7-974d-dbc3d80fe740 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:44.611668 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496330e5-3f40-44d7-974d-dbc3d80fe740 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:44.634486 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-496330e5-3f40-44d7-974d-dbc3d80fe740 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:44.635670 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-496330e5-3f40-44d7-974d-dbc3d80fe740 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:44.715186 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 167/336] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:44.895041 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39cafd2-3345-42e5-ae52-3779af6ec953 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:44.895730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39cafd2-3345-42e5-ae52-3779af6ec953 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:44.896209 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39cafd2-3345-42e5-ae52-3779af6ec953 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:44.898041 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 170/337] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:44.903526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4308cb-4ebd-4347-9e8c-798bcd4bc9e4 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:44.903813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4308cb-4ebd-4347-9e8c-798bcd4bc9e4 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:44.904088 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4308cb-4ebd-4347-9e8c-798bcd4bc9e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:44.925700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-8c4308cb-4ebd-4347-9e8c-798bcd4bc9e4 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:44.927451 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-8c4308cb-4ebd-4347-9e8c-798bcd4bc9e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:45.014118 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 168/338] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:45.165916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39cafd2-3345-42e5-ae52-3779af6ec953 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:45.285088 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39cafd2-3345-42e5-ae52-3779af6ec953 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:45.287174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:45.322216 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35373ab7-ad5b-4cae-9466-c2e88e6024ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:45.322851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35373ab7-ad5b-4cae-9466-c2e88e6024ed None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:45.323132 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35373ab7-ad5b-4cae-9466-c2e88e6024ed None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:45.323404 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35373ab7-ad5b-4cae-9466-c2e88e6024ed None admin] PATH_INFO: `/v3/roles/member` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:45.335135 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35373ab7-ad5b-4cae-9466-c2e88e6024ed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:45.340059 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-35373ab7-ad5b-4cae-9466-c2e88e6024ed None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:45.340059 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:45.340059 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:45.340059 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:45.340059 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:45.340059 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:45.340059 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:45.340059 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:45.340059 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:45.340059 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:45.340059 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:45.340059 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:45.340059 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:45.340059 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:45.340059 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:45.340059 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:45.340923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:45.340923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:45.340923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:45.340923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:45.340923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:45.340923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:45.340923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:45.340923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:45.340923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:45.340923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:45.340923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:45.340923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:45.340923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:45.340923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:45.340923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:45.340923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:45.342344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:45.342344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:45.342344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:45.342344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:45.342344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:45.342344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:45.342344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:45.342344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:45.342344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:45.342344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:45.342344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:45.342344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:06:45.342344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 171/339] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:45 2022] GET /identity/v3/roles/member => generated 84 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:45.347108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4308cb-4ebd-4347-9e8c-798bcd4bc9e4 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:45.442112 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:45.481896 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35373ab7-ad5b-4cae-9466-c2e88e6024ed None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:45.553804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4308cb-4ebd-4347-9e8c-798bcd4bc9e4 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:45.593022 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35373ab7-ad5b-4cae-9466-c2e88e6024ed None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:45.630330 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452afb45-9e0a-4ef7-b865-dd9188d03fe2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:45.630949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452afb45-9e0a-4ef7-b865-dd9188d03fe2 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:45.631215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452afb45-9e0a-4ef7-b865-dd9188d03fe2 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:45.631492 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452afb45-9e0a-4ef7-b865-dd9188d03fe2 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:45.641136 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8efd66c-de48-4580-9d5c-70b8c46dc13e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:45.641780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8efd66c-de48-4580-9d5c-70b8c46dc13e None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:45.642060 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8efd66c-de48-4580-9d5c-70b8c46dc13e None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:45.642335 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8efd66c-de48-4580-9d5c-70b8c46dc13e None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:45.643009 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8efd66c-de48-4580-9d5c-70b8c46dc13e None admin] RBAC: Adding query filter params (name=member) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:45.643177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452afb45-9e0a-4ef7-b865-dd9188d03fe2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:45.643340 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8efd66c-de48-4580-9d5c-70b8c46dc13e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:45.644670 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8efd66c-de48-4580-9d5c-70b8c46dc13e None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:45.649294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-452afb45-9e0a-4ef7-b865-dd9188d03fe2 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:06:45.649294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:45.649294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:45.649294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:45.649294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:45.649294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:45.649294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:45.649294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:45.649294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:45.649294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:45.649294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:45.649294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:45.649294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:45.649294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:45.649294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:45.649294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:45.650196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:45.650196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:45.650196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:45.650196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:45.650196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:45.650196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:45.650196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:45.650196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:45.650196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:45.650196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:45.650196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:45.650196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:45.650196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:45.650196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:45.650196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:45.650196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:45.651010 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:45.651010 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:45.651010 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:45.651010 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:45.651010 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:45.651010 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:45.651010 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:45.651010 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:45.651010 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:45.651010 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:45.651010 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:06:45.651010 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:06:45.652903 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 172/340] 10.222.0.26 () {56 vars in 1114 bytes} [Tue Sep 27 04:06:45 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:45.656991 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452afb45-9e0a-4ef7-b865-dd9188d03fe2 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:45.663066 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 169/341] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:45 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:45.668762 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8efd66c-de48-4580-9d5c-70b8c46dc13e None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:45.686122 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:45.793467 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452afb45-9e0a-4ef7-b865-dd9188d03fe2 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:45.855133 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:45.859537 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8efd66c-de48-4580-9d5c-70b8c46dc13e None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:45.864034 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07b2cdd-1c6d-46ad-a692-d362f219b01c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:45.864674 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07b2cdd-1c6d-46ad-a692-d362f219b01c None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:45.864948 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07b2cdd-1c6d-46ad-a692-d362f219b01c None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:45.865218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07b2cdd-1c6d-46ad-a692-d362f219b01c None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:45.865916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07b2cdd-1c6d-46ad-a692-d362f219b01c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:45.866218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07b2cdd-1c6d-46ad-a692-d362f219b01c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:45.867585 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07b2cdd-1c6d-46ad-a692-d362f219b01c None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:45.891367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 173/342] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:06:45 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:45.893520 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07b2cdd-1c6d-46ad-a692-d362f219b01c None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:45.920719 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57e1ffc-d676-42e6-a7fe-7d743d3f5f57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:45.921364 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57e1ffc-d676-42e6-a7fe-7d743d3f5f57 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:45.921637 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57e1ffc-d676-42e6-a7fe-7d743d3f5f57 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:45.921941 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57e1ffc-d676-42e6-a7fe-7d743d3f5f57 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:46.046488 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57e1ffc-d676-42e6-a7fe-7d743d3f5f57 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:46.047972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57e1ffc-d676-42e6-a7fe-7d743d3f5f57 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:46.096546 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:46.112843 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07b2cdd-1c6d-46ad-a692-d362f219b01c None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:46.155838 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6edd5d-ea9f-4443-8a5b-6fb01c926d9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:46.156461 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6edd5d-ea9f-4443-8a5b-6fb01c926d9f None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:46.156742 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6edd5d-ea9f-4443-8a5b-6fb01c926d9f None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:46.157019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6edd5d-ea9f-4443-8a5b-6fb01c926d9f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:46.216559 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 170/343] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 548 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 27 04:06:46.216886 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:46.221064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5400cfd0-03ed-480b-bf5f-9da1a19abca1 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:46.221410 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5400cfd0-03ed-480b-bf5f-9da1a19abca1 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:46.221767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5400cfd0-03ed-480b-bf5f-9da1a19abca1 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:46.223867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 171/344] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:46 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:46.225508 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5400cfd0-03ed-480b-bf5f-9da1a19abca1 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:46.276213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6edd5d-ea9f-4443-8a5b-6fb01c926d9f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:46.277666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6edd5d-ea9f-4443-8a5b-6fb01c926d9f None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:46.372065 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5400cfd0-03ed-480b-bf5f-9da1a19abca1 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:46.384290 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:46.410532 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 174/345] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 517 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:06:46.413259 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9f8a86-bd6b-4d97-adc2-17222e7e490e None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:46.413736 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9f8a86-bd6b-4d97-adc2-17222e7e490e None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:46.414331 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9f8a86-bd6b-4d97-adc2-17222e7e490e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:46.424522 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1e7567-8d43-46ff-93a2-ba48b34536e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:46.425368 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1e7567-8d43-46ff-93a2-ba48b34536e7 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:46.425547 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1e7567-8d43-46ff-93a2-ba48b34536e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:46.425804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1e7567-8d43-46ff-93a2-ba48b34536e7 None admin] PATH_INFO: `/v3/projects/146fd1189c56413c9d17cd16d1a974d3` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:46.427756 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1e7567-8d43-46ff-93a2-ba48b34536e7 None admin] RBAC: Authorizing `identity:get_project(project_id=146fd1189c56413c9d17cd16d1a974d3)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:46.428867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1e7567-8d43-46ff-93a2-ba48b34536e7 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:46.430577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 172/346] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:46 2022] GET /identity/v3/projects/146fd1189c56413c9d17cd16d1a974d3 => generated 298 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:46.431345 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1e7567-8d43-46ff-93a2-ba48b34536e7 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:46.439986 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-bc9f8a86-bd6b-4d97-adc2-17222e7e490e None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:46.441436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-bc9f8a86-bd6b-4d97-adc2-17222e7e490e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:46.521580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 175/347] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:46.522799 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9f8a86-bd6b-4d97-adc2-17222e7e490e None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:46.572396 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:46.650033 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1e7567-8d43-46ff-93a2-ba48b34536e7 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:46.721099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71721f7-ce0b-43cd-94b4-87ae3ac0f2c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:46.721758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71721f7-ce0b-43cd-94b4-87ae3ac0f2c0 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:46.722045 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71721f7-ce0b-43cd-94b4-87ae3ac0f2c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:46.722321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71721f7-ce0b-43cd-94b4-87ae3ac0f2c0 None admin] PATH_INFO: `/v3/projects/146fd1189c56413c9d17cd16d1a974d3` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:46.723451 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71721f7-ce0b-43cd-94b4-87ae3ac0f2c0 None admin] RBAC: Authorizing `identity:get_project(project_id=146fd1189c56413c9d17cd16d1a974d3)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:46.723882 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9f8a86-bd6b-4d97-adc2-17222e7e490e None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:46.724933 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71721f7-ce0b-43cd-94b4-87ae3ac0f2c0 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:46.726598 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 173/348] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:46 2022] GET /identity/v3/projects/146fd1189c56413c9d17cd16d1a974d3 => generated 298 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:46.733026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71721f7-ce0b-43cd-94b4-87ae3ac0f2c0 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:46.745361 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:46.776595 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f034e0a2-c639-44a9-a17f-1285eb728e29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:46.777579 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f034e0a2-c639-44a9-a17f-1285eb728e29 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:46.778094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f034e0a2-c639-44a9-a17f-1285eb728e29 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:46.778590 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f034e0a2-c639-44a9-a17f-1285eb728e29 None admin] PATH_INFO: `/v3/groups/1b59b23587524103b43265b13a5157d1` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:46.790772 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f034e0a2-c639-44a9-a17f-1285eb728e29 None admin] RBAC: Authorizing `identity:get_group(group_id=1b59b23587524103b43265b13a5157d1)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:46.792530 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f034e0a2-c639-44a9-a17f-1285eb728e29 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:46.794575 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 176/349] 10.222.0.26 () {56 vars in 1180 bytes} [Tue Sep 27 04:06:46 2022] GET /identity/v3/groups/1b59b23587524103b43265b13a5157d1 => generated 226 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:46.796110 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9a8915-f306-4c5b-bfcb-67947ffe78b0 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:46.796928 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9a8915-f306-4c5b-bfcb-67947ffe78b0 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:46.797398 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9a8915-f306-4c5b-bfcb-67947ffe78b0 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:46.798829 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 177/350] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:46.800033 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c427fd0b-d9d6-4c83-9708-d93f87b2923d None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:46.800527 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c427fd0b-d9d6-4c83-9708-d93f87b2923d None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:46.801008 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c427fd0b-d9d6-4c83-9708-d93f87b2923d None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:46.809107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 178/351] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:46.809107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c427fd0b-d9d6-4c83-9708-d93f87b2923d None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:46.903414 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71721f7-ce0b-43cd-94b4-87ae3ac0f2c0 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:46.933150 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c427fd0b-d9d6-4c83-9708-d93f87b2923d None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:46.948393 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6f95c4-c135-48f1-b650-f323b310d4f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:46.949035 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6f95c4-c135-48f1-b650-f323b310d4f6 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:46.949307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6f95c4-c135-48f1-b650-f323b310d4f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:46.949577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6f95c4-c135-48f1-b650-f323b310d4f6 None admin] PATH_INFO: `/v3/groups/1b59b23587524103b43265b13a5157d1` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:46.951167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6f95c4-c135-48f1-b650-f323b310d4f6 None admin] RBAC: Authorizing `identity:get_group(group_id=1b59b23587524103b43265b13a5157d1)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:46.952597 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6f95c4-c135-48f1-b650-f323b310d4f6 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:46.955116 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 174/352] 10.222.0.26 () {56 vars in 1180 bytes} [Tue Sep 27 04:06:46 2022] GET /identity/v3/groups/1b59b23587524103b43265b13a5157d1 => generated 226 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:46.956367 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7668e6e3-5c23-46f1-abda-07a1b596b77c None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:46.956646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7668e6e3-5c23-46f1-abda-07a1b596b77c None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:46.956912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7668e6e3-5c23-46f1-abda-07a1b596b77c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:46.964816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:46.976305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-7668e6e3-5c23-46f1-abda-07a1b596b77c None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:46.977716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-7668e6e3-5c23-46f1-abda-07a1b596b77c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:46.978263 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15da35e1-999f-4bca-b1bd-2073e2957652 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:46.978997 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15da35e1-999f-4bca-b1bd-2073e2957652 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:46.979276 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15da35e1-999f-4bca-b1bd-2073e2957652 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:46.979549 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15da35e1-999f-4bca-b1bd-2073e2957652 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:46.980365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15da35e1-999f-4bca-b1bd-2073e2957652 None admin] RBAC: Adding query filter params (group.id=1b59b23587524103b43265b13a5157d1, role.id=e3ce5a3697ec4292a446366d3caadeea, scope.project.id=146fd1189c56413c9d17cd16d1a974d3) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:46.980676 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15da35e1-999f-4bca-b1bd-2073e2957652 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:46.982127 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15da35e1-999f-4bca-b1bd-2073e2957652 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:46.992049 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 179/353] 10.222.0.26 () {56 vars in 1376 bytes} [Tue Sep 27 04:06:46 2022] GET /identity/v3/role_assignments?group.id=1b59b23587524103b43265b13a5157d1&scope.project.id=146fd1189c56413c9d17cd16d1a974d3&role.id=e3ce5a3697ec4292a446366d3caadeea => generated 260 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:46.994819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d927f4c1-6e15-4047-a5a3-2021859eb679 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:46.994819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d927f4c1-6e15-4047-a5a3-2021859eb679 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:46.994819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d927f4c1-6e15-4047-a5a3-2021859eb679 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:47.019044 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-d927f4c1-6e15-4047-a5a3-2021859eb679 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:47.020290 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-d927f4c1-6e15-4047-a5a3-2021859eb679 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:47.064654 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 175/354] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:47.065555 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7668e6e3-5c23-46f1-abda-07a1b596b77c None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:47.076359 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:47.109391 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 180/355] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:47.115399 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d927f4c1-6e15-4047-a5a3-2021859eb679 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:47.251533 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d927f4c1-6e15-4047-a5a3-2021859eb679 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:47.265547 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7668e6e3-5c23-46f1-abda-07a1b596b77c None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:47.285294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589dfe36-2b49-4c66-8023-60fa92ed5619 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:47.285945 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589dfe36-2b49-4c66-8023-60fa92ed5619 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:47.286768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589dfe36-2b49-4c66-8023-60fa92ed5619 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:47.287040 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589dfe36-2b49-4c66-8023-60fa92ed5619 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:47.287774 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589dfe36-2b49-4c66-8023-60fa92ed5619 None admin] RBAC: Adding query filter params (group.id=1b59b23587524103b43265b13a5157d1, role.id=2d9f391485604d0b952b7c844c3a3266, scope.project.id=146fd1189c56413c9d17cd16d1a974d3) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:47.288099 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589dfe36-2b49-4c66-8023-60fa92ed5619 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:47.289495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589dfe36-2b49-4c66-8023-60fa92ed5619 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:47.295409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 181/356] 10.222.0.26 () {56 vars in 1376 bytes} [Tue Sep 27 04:06:47 2022] GET /identity/v3/role_assignments?group.id=1b59b23587524103b43265b13a5157d1&scope.project.id=146fd1189c56413c9d17cd16d1a974d3&role.id=2d9f391485604d0b952b7c844c3a3266 => generated 260 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:47.338657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964ab0c7-b9f4-404d-8c4e-3036eb18db20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:47.339702 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964ab0c7-b9f4-404d-8c4e-3036eb18db20 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:47.340174 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964ab0c7-b9f4-404d-8c4e-3036eb18db20 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:47.340637 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964ab0c7-b9f4-404d-8c4e-3036eb18db20 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:47.343396 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:47.359931 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964ab0c7-b9f4-404d-8c4e-3036eb18db20 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:47.368473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-964ab0c7-b9f4-404d-8c4e-3036eb18db20 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:47.368473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:47.368473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:47.368473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:47.368473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:47.368473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:47.368473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:47.368473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:47.368473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:47.368473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:47.368473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:47.368473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:47.368473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:47.368473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:47.368473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:47.368473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:47.369461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:47.369461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:47.369461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:47.369461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:47.369461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:47.369461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:47.369461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:47.369461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:47.369461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:47.369461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:47.369461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:47.369461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:47.369461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:47.369461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:47.369461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:47.369461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:47.370321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:47.370321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:47.370321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:47.370321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:47.370321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:47.370321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:47.370321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:47.370321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:47.370321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:47.370321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:47.370321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:47.370321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:06:47.370321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 176/357] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:47 2022] GET /identity/v3/roles/member => generated 84 bytes in 304 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:47.375837 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589dfe36-2b49-4c66-8023-60fa92ed5619 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:47.439070 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:47.522214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589dfe36-2b49-4c66-8023-60fa92ed5619 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:47.556926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964ab0c7-b9f4-404d-8c4e-3036eb18db20 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:47.561181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9037e384-64f2-411c-99bc-ef4480458401 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:47.562191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9037e384-64f2-411c-99bc-ef4480458401 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:47.562521 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9037e384-64f2-411c-99bc-ef4480458401 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:47.562835 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9037e384-64f2-411c-99bc-ef4480458401 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:47.563886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9037e384-64f2-411c-99bc-ef4480458401 None admin] RBAC: Adding query filter params (name=member) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:47.563886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9037e384-64f2-411c-99bc-ef4480458401 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:47.565207 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9037e384-64f2-411c-99bc-ef4480458401 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:47.577884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 182/358] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:47 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:47.580389 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9037e384-64f2-411c-99bc-ef4480458401 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:47.704037 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:47.710641 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9037e384-64f2-411c-99bc-ef4480458401 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:47.769374 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964ab0c7-b9f4-404d-8c4e-3036eb18db20 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:47.777048 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a2576f-4339-458c-be29-5f308e638c96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:47.777605 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a2576f-4339-458c-be29-5f308e638c96 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:47.777950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a2576f-4339-458c-be29-5f308e638c96 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:47.778233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a2576f-4339-458c-be29-5f308e638c96 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:47.817001 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178cdb55-7a0b-46e6-89bb-3626068ec11d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:47.817675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178cdb55-7a0b-46e6-89bb-3626068ec11d None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:47.817996 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178cdb55-7a0b-46e6-89bb-3626068ec11d None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:47.818269 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178cdb55-7a0b-46e6-89bb-3626068ec11d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:47.834039 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178cdb55-7a0b-46e6-89bb-3626068ec11d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:47.839048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-178cdb55-7a0b-46e6-89bb-3626068ec11d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:06:47.839048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:47.839048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:47.839048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:47.839048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:47.839048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:47.839048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:47.839048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:47.839048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:47.839048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:47.839048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:47.839048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:47.839048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:47.839048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:47.839048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:47.839048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:47.839943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:47.839943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:47.839943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:47.839943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:47.839943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:47.839943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:47.839943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:47.839943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:47.839943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:47.839943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:47.839943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:47.839943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:47.839943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:47.839943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:47.839943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:47.839943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:47.840743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:47.840743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:47.840743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:47.840743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:47.840743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:47.840743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:47.840743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:47.840743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:47.840743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:47.840743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:47.840743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:06:47.840743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:06:47.840743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 177/359] 10.222.0.26 () {56 vars in 1114 bytes} [Tue Sep 27 04:06:47 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:47.841464 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178cdb55-7a0b-46e6-89bb-3626068ec11d None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:47.866567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a2576f-4339-458c-be29-5f308e638c96 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:47.868024 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a2576f-4339-458c-be29-5f308e638c96 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:47.912661 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:47.970349 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178cdb55-7a0b-46e6-89bb-3626068ec11d None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:47.992871 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:47.999785 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 183/360] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 27 04:06:48.001792 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39e57ef-ec82-4e50-821b-c2efff02ab1b None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:48.003168 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39e57ef-ec82-4e50-821b-c2efff02ab1b None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:48.003595 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39e57ef-ec82-4e50-821b-c2efff02ab1b None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:48.005839 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 184/361] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:48 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:48.006573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39e57ef-ec82-4e50-821b-c2efff02ab1b None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:48.012782 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be09fcb5-785d-4dd1-b7b1-89c8a7e2170f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:48.013420 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be09fcb5-785d-4dd1-b7b1-89c8a7e2170f None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:48.013693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be09fcb5-785d-4dd1-b7b1-89c8a7e2170f None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:48.014008 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be09fcb5-785d-4dd1-b7b1-89c8a7e2170f None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:48.026826 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be09fcb5-785d-4dd1-b7b1-89c8a7e2170f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:48.032553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-be09fcb5-785d-4dd1-b7b1-89c8a7e2170f None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 04:06:48.032553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:48.032553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:48.032553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:48.032553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:48.032553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:48.032553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:48.032553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:48.032553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:48.032553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:48.032553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:48.032553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:48.032553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:48.032553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:48.032553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:48.032553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:48.034629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:48.034629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:48.034629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:48.034629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:48.034629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:48.034629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:48.034629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:48.034629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:48.034629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:48.034629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:48.034629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:48.034629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:48.034629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:48.034629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:48.034629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:48.034629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:48.036592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:48.036592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:48.036592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:48.036592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:48.036592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:48.036592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:48.036592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:48.036592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:48.036592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:48.036592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:48.036592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 04:06:48.036592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:06:48.036592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 178/362] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:47 2022] GET /identity/v3/roles/reader => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:48.036592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be09fcb5-785d-4dd1-b7b1-89c8a7e2170f None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:48.146495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39e57ef-ec82-4e50-821b-c2efff02ab1b None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:48.151010 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be09fcb5-785d-4dd1-b7b1-89c8a7e2170f None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:48.169728 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:48.198824 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69ba561-6295-4fc9-8fc3-1c5eb9314454 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:48.199433 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69ba561-6295-4fc9-8fc3-1c5eb9314454 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:48.199709 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69ba561-6295-4fc9-8fc3-1c5eb9314454 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:48.201287 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69ba561-6295-4fc9-8fc3-1c5eb9314454 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:48.201287 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69ba561-6295-4fc9-8fc3-1c5eb9314454 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:48.201287 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69ba561-6295-4fc9-8fc3-1c5eb9314454 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:48.202309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69ba561-6295-4fc9-8fc3-1c5eb9314454 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:48.203085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d90c36-9eec-43db-ae37-c6352599b2d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:48.204007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d90c36-9eec-43db-ae37-c6352599b2d5 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:48.204455 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d90c36-9eec-43db-ae37-c6352599b2d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:48.204938 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d90c36-9eec-43db-ae37-c6352599b2d5 None admin] PATH_INFO: `/v3/projects/bbe6cda6a9564434b77be066e776bd55` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:48.206283 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d90c36-9eec-43db-ae37-c6352599b2d5 None admin] RBAC: Authorizing `identity:get_project(project_id=bbe6cda6a9564434b77be066e776bd55)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:48.208128 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d90c36-9eec-43db-ae37-c6352599b2d5 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:48.210548 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 179/363] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:48 2022] GET /identity/v3/projects/bbe6cda6a9564434b77be066e776bd55 => generated 302 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:48.215793 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 185/364] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:06:48 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:48.215793 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69ba561-6295-4fc9-8fc3-1c5eb9314454 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:48.215984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e52a97-b604-4cda-985f-294861a9761d None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:48.215984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e52a97-b604-4cda-985f-294861a9761d None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:48.215984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e52a97-b604-4cda-985f-294861a9761d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:48.243374 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-54e52a97-b604-4cda-985f-294861a9761d None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:48.244058 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-54e52a97-b604-4cda-985f-294861a9761d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:48.281234 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:48.325517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69ba561-6295-4fc9-8fc3-1c5eb9314454 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:48.325869 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 180/365] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:48.326721 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e52a97-b604-4cda-985f-294861a9761d None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:48.374456 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef68d11-8264-4464-952a-dcc186bbe14c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:48.375423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef68d11-8264-4464-952a-dcc186bbe14c None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:48.375904 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef68d11-8264-4464-952a-dcc186bbe14c None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:48.376359 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef68d11-8264-4464-952a-dcc186bbe14c None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:48.377213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef68d11-8264-4464-952a-dcc186bbe14c None admin] RBAC: Adding query filter params (name=reader) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:48.377986 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef68d11-8264-4464-952a-dcc186bbe14c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:48.379539 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef68d11-8264-4464-952a-dcc186bbe14c None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:48.389145 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 186/366] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:48 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:48.394583 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef68d11-8264-4464-952a-dcc186bbe14c None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:48.438728 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e52a97-b604-4cda-985f-294861a9761d None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:48.453376 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:48.477124 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11934913-6c9c-406d-b4e6-74eeddc0e3f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:48.478098 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11934913-6c9c-406d-b4e6-74eeddc0e3f3 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:48.478560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11934913-6c9c-406d-b4e6-74eeddc0e3f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:48.478971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11934913-6c9c-406d-b4e6-74eeddc0e3f3 None admin] PATH_INFO: `/v3/groups/1b59b23587524103b43265b13a5157d1` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:48.480221 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11934913-6c9c-406d-b4e6-74eeddc0e3f3 None admin] RBAC: Authorizing `identity:get_group(group_id=1b59b23587524103b43265b13a5157d1)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:48.481805 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11934913-6c9c-406d-b4e6-74eeddc0e3f3 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:48.483526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 181/367] 10.222.0.26 () {56 vars in 1180 bytes} [Tue Sep 27 04:06:48 2022] GET /identity/v3/groups/1b59b23587524103b43265b13a5157d1 => generated 226 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:48.484475 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11934913-6c9c-406d-b4e6-74eeddc0e3f3 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:48.545923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef68d11-8264-4464-952a-dcc186bbe14c None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:48.578931 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:48.595703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11934913-6c9c-406d-b4e6-74eeddc0e3f3 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:48.617596 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3ad9a1-f1d6-4cf4-85a6-1a3d7bda78ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:48.618412 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3ad9a1-f1d6-4cf4-85a6-1a3d7bda78ac None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:48.618807 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3ad9a1-f1d6-4cf4-85a6-1a3d7bda78ac None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:48.619179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3ad9a1-f1d6-4cf4-85a6-1a3d7bda78ac None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:48.636392 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c971f366-4056-40e1-8d74-4c0786a3d02d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:48.637755 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c971f366-4056-40e1-8d74-4c0786a3d02d None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:48.638237 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c971f366-4056-40e1-8d74-4c0786a3d02d None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:48.638646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c971f366-4056-40e1-8d74-4c0786a3d02d None admin] PATH_INFO: `/v3/users/16a4e626417f43e6a60ac07c6e263212` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:48.640060 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c971f366-4056-40e1-8d74-4c0786a3d02d None admin] RBAC: Authorizing `identity:get_user(user_id=16a4e626417f43e6a60ac07c6e263212)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:48.641709 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c971f366-4056-40e1-8d74-4c0786a3d02d None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:48.643574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 182/368] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:06:48 2022] GET /identity/v3/users/16a4e626417f43e6a60ac07c6e263212 => generated 293 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:48.644611 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c971f366-4056-40e1-8d74-4c0786a3d02d None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:48.700036 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:48.751642 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c971f366-4056-40e1-8d74-4c0786a3d02d None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:48.789134 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3ad9a1-f1d6-4cf4-85a6-1a3d7bda78ac None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:48.793051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3ad9a1-f1d6-4cf4-85a6-1a3d7bda78ac None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:48.795650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c45a646-d6fd-4e89-962e-5e55d5a5e24b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:48.796825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c45a646-d6fd-4e89-962e-5e55d5a5e24b None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:48.797294 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c45a646-d6fd-4e89-962e-5e55d5a5e24b None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:48.797799 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c45a646-d6fd-4e89-962e-5e55d5a5e24b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:48.798727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c45a646-d6fd-4e89-962e-5e55d5a5e24b None admin] RBAC: Adding query filter params (group.id=1b59b23587524103b43265b13a5157d1, role.id=e3ce5a3697ec4292a446366d3caadeea, scope.project.id=bbe6cda6a9564434b77be066e776bd55) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:48.799222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c45a646-d6fd-4e89-962e-5e55d5a5e24b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:48.801210 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c45a646-d6fd-4e89-962e-5e55d5a5e24b None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:48.813253 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 183/369] 10.222.0.26 () {56 vars in 1376 bytes} [Tue Sep 27 04:06:48 2022] GET /identity/v3/role_assignments?group.id=1b59b23587524103b43265b13a5157d1&scope.project.id=bbe6cda6a9564434b77be066e776bd55&role.id=e3ce5a3697ec4292a446366d3caadeea => generated 260 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:48.814368 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c45a646-d6fd-4e89-962e-5e55d5a5e24b None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:48.931309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 187/370] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 541 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:06:48.932141 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3ad9a1-f1d6-4cf4-85a6-1a3d7bda78ac None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:48.940327 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c45a646-d6fd-4e89-962e-5e55d5a5e24b None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:48.955460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:48.977390 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:48.982126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c87f865-40fb-46d7-91ec-16874c5272a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:48.984142 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c87f865-40fb-46d7-91ec-16874c5272a0 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:48.984620 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c87f865-40fb-46d7-91ec-16874c5272a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:48.985146 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c87f865-40fb-46d7-91ec-16874c5272a0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:48.987789 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c87f865-40fb-46d7-91ec-16874c5272a0 None admin] RBAC: Adding query filter params (role.id=9b729a0f423e47f7bc562c000780ef7b, user.id=16a4e626417f43e6a60ac07c6e263212, scope.system=all) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:48.988391 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c87f865-40fb-46d7-91ec-16874c5272a0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:48.990201 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c87f865-40fb-46d7-91ec-16874c5272a0 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:49.005862 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 184/371] 10.222.0.26 () {56 vars in 1308 bytes} [Tue Sep 27 04:06:48 2022] GET /identity/v3/role_assignments?user.id=16a4e626417f43e6a60ac07c6e263212&scope.system=all&role.id=9b729a0f423e47f7bc562c000780ef7b => generated 510 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:49.007062 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c87f865-40fb-46d7-91ec-16874c5272a0 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:49.101237 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3ad9a1-f1d6-4cf4-85a6-1a3d7bda78ac None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:49.124281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c87f865-40fb-46d7-91ec-16874c5272a0 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:49.166564 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78c7fcd-4a9e-4b9a-96a4-622b0607d226 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:49.167569 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78c7fcd-4a9e-4b9a-96a4-622b0607d226 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:49.168041 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78c7fcd-4a9e-4b9a-96a4-622b0607d226 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:49.168508 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78c7fcd-4a9e-4b9a-96a4-622b0607d226 None admin] PATH_INFO: `/v3/projects/bbe6cda6a9564434b77be066e776bd55` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:49.169961 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78c7fcd-4a9e-4b9a-96a4-622b0607d226 None admin] RBAC: Authorizing `identity:get_project(project_id=bbe6cda6a9564434b77be066e776bd55)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:49.171649 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78c7fcd-4a9e-4b9a-96a4-622b0607d226 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:49.174579 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 185/372] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:49 2022] GET /identity/v3/projects/bbe6cda6a9564434b77be066e776bd55 => generated 302 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:49.182969 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78c7fcd-4a9e-4b9a-96a4-622b0607d226 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:49.202705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a22e13e-f3ca-4227-9579-4aceed9ae7e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:49.203394 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a22e13e-f3ca-4227-9579-4aceed9ae7e3 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:49.203679 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a22e13e-f3ca-4227-9579-4aceed9ae7e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:49.203955 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a22e13e-f3ca-4227-9579-4aceed9ae7e3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:49.227950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a22e13e-f3ca-4227-9579-4aceed9ae7e3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:49.233131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-7a22e13e-f3ca-4227-9579-4aceed9ae7e3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:06:49.233131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:49.233131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:49.233131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:49.233131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:49.233131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:49.233131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:49.233131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:49.233131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:49.233131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:49.233131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:49.233131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:49.233131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:49.233131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:49.233131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:49.233131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:49.234226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:49.234226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:49.234226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:49.234226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:49.234226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:49.234226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:49.234226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:49.234226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:49.234226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:49.234226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:49.234226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:49.234226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:49.234226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:49.234226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:49.234226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:49.234226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:49.235017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:49.235017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:49.235017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:49.235017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:49.235017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:49.235017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:49.235017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:49.235017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:49.235017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:49.235017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:49.235017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:06:49.235017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:06:49.235017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 188/373] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:06:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 303 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:49.244341 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a22e13e-f3ca-4227-9579-4aceed9ae7e3 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:49.261295 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:49.373841 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78c7fcd-4a9e-4b9a-96a4-622b0607d226 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:49.411787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:49.446666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a22e13e-f3ca-4227-9579-4aceed9ae7e3 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:49.450731 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbf5e60-15c9-4b8d-b504-a1ffb265bda9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:49.451378 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbf5e60-15c9-4b8d-b504-a1ffb265bda9 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:49.451659 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbf5e60-15c9-4b8d-b504-a1ffb265bda9 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:49.452072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbf5e60-15c9-4b8d-b504-a1ffb265bda9 None admin] PATH_INFO: `/v3/groups/1b59b23587524103b43265b13a5157d1` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:49.454853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbf5e60-15c9-4b8d-b504-a1ffb265bda9 None admin] RBAC: Authorizing `identity:get_group(group_id=1b59b23587524103b43265b13a5157d1)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:49.456314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbf5e60-15c9-4b8d-b504-a1ffb265bda9 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:49.457945 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 186/374] 10.222.0.26 () {56 vars in 1180 bytes} [Tue Sep 27 04:06:49 2022] GET /identity/v3/groups/1b59b23587524103b43265b13a5157d1 => generated 226 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:49.466003 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbf5e60-15c9-4b8d-b504-a1ffb265bda9 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:49.503275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305be4b2-2273-4f24-a5f5-0a86e560184b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:49.505218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305be4b2-2273-4f24-a5f5-0a86e560184b None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:49.505510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305be4b2-2273-4f24-a5f5-0a86e560184b None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:49.505806 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305be4b2-2273-4f24-a5f5-0a86e560184b None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:49.507914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305be4b2-2273-4f24-a5f5-0a86e560184b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:49.508227 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305be4b2-2273-4f24-a5f5-0a86e560184b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:49.510215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305be4b2-2273-4f24-a5f5-0a86e560184b None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:49.525512 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 189/375] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:49.532013 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305be4b2-2273-4f24-a5f5-0a86e560184b None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:49.665536 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305be4b2-2273-4f24-a5f5-0a86e560184b None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:49.673991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbf5e60-15c9-4b8d-b504-a1ffb265bda9 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:49.683692 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:49.709354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022fe5f3-dbfb-4a36-b5f3-3d7132a20305 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:49.709907 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022fe5f3-dbfb-4a36-b5f3-3d7132a20305 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:49.710187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022fe5f3-dbfb-4a36-b5f3-3d7132a20305 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:49.710459 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022fe5f3-dbfb-4a36-b5f3-3d7132a20305 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:49.732553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:49.747451 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a04e911-4366-486a-b905-99c8a3c96d95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:49.748397 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a04e911-4366-486a-b905-99c8a3c96d95 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:49.748872 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a04e911-4366-486a-b905-99c8a3c96d95 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:49.749346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a04e911-4366-486a-b905-99c8a3c96d95 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:49.750362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a04e911-4366-486a-b905-99c8a3c96d95 None admin] RBAC: Adding query filter params (group.id=1b59b23587524103b43265b13a5157d1, role.id=2d9f391485604d0b952b7c844c3a3266, scope.project.id=bbe6cda6a9564434b77be066e776bd55) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:49.750901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a04e911-4366-486a-b905-99c8a3c96d95 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:49.752807 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a04e911-4366-486a-b905-99c8a3c96d95 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:49.768320 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 187/376] 10.222.0.26 () {56 vars in 1376 bytes} [Tue Sep 27 04:06:49 2022] GET /identity/v3/role_assignments?group.id=1b59b23587524103b43265b13a5157d1&scope.project.id=bbe6cda6a9564434b77be066e776bd55&role.id=2d9f391485604d0b952b7c844c3a3266 => generated 260 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:49.793093 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022fe5f3-dbfb-4a36-b5f3-3d7132a20305 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:49.794550 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022fe5f3-dbfb-4a36-b5f3-3d7132a20305 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:49.907835 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 190/377] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 27 04:06:49.914253 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a04e911-4366-486a-b905-99c8a3c96d95 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:50.032520 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bcf6d24-7939-4719-8b69-198f66cbbd43 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:50.032848 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bcf6d24-7939-4719-8b69-198f66cbbd43 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:50.033069 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bcf6d24-7939-4719-8b69-198f66cbbd43 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:50.034673 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 191/378] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:06:50.040420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2510561-cb98-4dd5-965d-7566eab6b1dd None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:50.040698 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2510561-cb98-4dd5-965d-7566eab6b1dd None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:50.041157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2510561-cb98-4dd5-965d-7566eab6b1dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:50.066099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a04e911-4366-486a-b905-99c8a3c96d95 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:50.068811 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-d2510561-cb98-4dd5-965d-7566eab6b1dd None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:50.070859 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-d2510561-cb98-4dd5-965d-7566eab6b1dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:50.109856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb4ded8-d77f-431e-b7b2-96762c2bfca4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:50.110492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb4ded8-d77f-431e-b7b2-96762c2bfca4 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:50.110768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb4ded8-d77f-431e-b7b2-96762c2bfca4 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:50.111042 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb4ded8-d77f-431e-b7b2-96762c2bfca4 None admin] PATH_INFO: `/v3/projects/3cc5c8e99cb64893a8d035d6aa4f3eb6` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:50.112163 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb4ded8-d77f-431e-b7b2-96762c2bfca4 None admin] RBAC: Authorizing `identity:get_project(project_id=3cc5c8e99cb64893a8d035d6aa4f3eb6)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:50.113596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb4ded8-d77f-431e-b7b2-96762c2bfca4 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:50.116790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 188/379] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:49 2022] GET /identity/v3/projects/3cc5c8e99cb64893a8d035d6aa4f3eb6 => generated 344 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:50.122758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb4ded8-d77f-431e-b7b2-96762c2bfca4 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:50.127413 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:50.165518 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 192/380] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:06:50.191241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:50.277237 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb4ded8-d77f-431e-b7b2-96762c2bfca4 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:50.322291 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f96f1a-6e17-4b78-a92e-f4af91fa2627 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:50.322476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f96f1a-6e17-4b78-a92e-f4af91fa2627 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:50.322552 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba683b15-55be-41aa-89b5-aa40360680a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:50.322788 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f96f1a-6e17-4b78-a92e-f4af91fa2627 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:50.323157 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba683b15-55be-41aa-89b5-aa40360680a3 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:50.323436 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba683b15-55be-41aa-89b5-aa40360680a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:50.323707 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba683b15-55be-41aa-89b5-aa40360680a3 None admin] PATH_INFO: `/v3/groups/4564949965a9439ca5dae605a38a0650` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:50.324322 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 193/381] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:50.329596 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de577f9-ca28-454a-8339-b264dc3f69dd None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:50.329948 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de577f9-ca28-454a-8339-b264dc3f69dd None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:50.330287 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de577f9-ca28-454a-8339-b264dc3f69dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:50.335434 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba683b15-55be-41aa-89b5-aa40360680a3 None admin] RBAC: Authorizing `identity:get_group(group_id=4564949965a9439ca5dae605a38a0650)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:50.336887 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba683b15-55be-41aa-89b5-aa40360680a3 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:50.338857 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 189/382] 10.222.0.26 () {56 vars in 1180 bytes} [Tue Sep 27 04:06:50 2022] GET /identity/v3/groups/4564949965a9439ca5dae605a38a0650 => generated 229 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:50.344470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba683b15-55be-41aa-89b5-aa40360680a3 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:50.356757 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-9de577f9-ca28-454a-8339-b264dc3f69dd None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:50.358006 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-9de577f9-ca28-454a-8339-b264dc3f69dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:50.450770 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 194/383] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:06:50.531331 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba683b15-55be-41aa-89b5-aa40360680a3 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:50.532563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de577f9-ca28-454a-8339-b264dc3f69dd None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:50.533129 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:50.601016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee2423d-f320-4118-9c75-faeae093648a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:50.601973 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee2423d-f320-4118-9c75-faeae093648a None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:50.607576 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee2423d-f320-4118-9c75-faeae093648a None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:50.608055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee2423d-f320-4118-9c75-faeae093648a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:50.608971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee2423d-f320-4118-9c75-faeae093648a None admin] RBAC: Adding query filter params (group.id=4564949965a9439ca5dae605a38a0650, role.id=e6a31562a18649f0809202c36a8e18ef, scope.project.id=3cc5c8e99cb64893a8d035d6aa4f3eb6) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:50.609503 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee2423d-f320-4118-9c75-faeae093648a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:50.611141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee2423d-f320-4118-9c75-faeae093648a None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:50.619485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 190/384] 10.222.0.26 () {56 vars in 1376 bytes} [Tue Sep 27 04:06:50 2022] GET /identity/v3/role_assignments?group.id=4564949965a9439ca5dae605a38a0650&scope.project.id=3cc5c8e99cb64893a8d035d6aa4f3eb6&role.id=e6a31562a18649f0809202c36a8e18ef => generated 260 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:50.693281 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de577f9-ca28-454a-8339-b264dc3f69dd None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:50.695807 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:50.742022 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c147c2-4d41-4699-af86-8d86e55a2554 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:50.742685 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c147c2-4d41-4699-af86-8d86e55a2554 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:50.742966 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c147c2-4d41-4699-af86-8d86e55a2554 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:50.743239 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c147c2-4d41-4699-af86-8d86e55a2554 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:50.756742 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c147c2-4d41-4699-af86-8d86e55a2554 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:50.766321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-66c147c2-4d41-4699-af86-8d86e55a2554 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:50.766321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:50.766321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:50.766321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:50.766321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:50.766321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:50.766321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:50.766321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:50.766321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:50.766321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:50.766321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:50.766321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:50.766321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:50.766321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:50.766321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:50.766321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:50.767264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:50.767264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:50.767264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:50.767264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:50.767264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:50.767264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:50.767264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:50.767264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:50.767264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:50.767264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:50.767264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:50.767264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:50.767264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:50.767264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:50.767264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:50.767264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:50.768506 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:50.768506 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:50.768506 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:50.768506 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:50.768506 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:50.768506 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:50.768506 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:50.768506 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:50.768506 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:50.768506 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:50.768506 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:50.768506 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:06:50.768506 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 195/385] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:50 2022] GET /identity/v3/roles/member => generated 84 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:06:50.775456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee2423d-f320-4118-9c75-faeae093648a None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:50.798582 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c147c2-4d41-4699-af86-8d86e55a2554 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:50.929531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c147c2-4d41-4699-af86-8d86e55a2554 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:50.980671 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee2423d-f320-4118-9c75-faeae093648a None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:50.986041 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f940f5bb-d8ea-449f-aaaa-4c39b86e9026 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:50.986679 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f940f5bb-d8ea-449f-aaaa-4c39b86e9026 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:50.986960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f940f5bb-d8ea-449f-aaaa-4c39b86e9026 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:50.987232 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f940f5bb-d8ea-449f-aaaa-4c39b86e9026 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:51.007520 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f940f5bb-d8ea-449f-aaaa-4c39b86e9026 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:51.012913 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-f940f5bb-d8ea-449f-aaaa-4c39b86e9026 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:06:51.012913 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:51.012913 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:51.012913 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:51.012913 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:51.012913 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:51.012913 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:51.012913 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:51.012913 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:51.012913 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:51.012913 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:51.012913 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:51.012913 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:51.012913 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:51.012913 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:51.012913 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:51.013891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:51.013891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:51.013891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:51.013891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:51.013891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:51.013891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:51.013891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:51.013891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:51.013891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:51.013891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:51.013891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:51.013891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:51.013891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:51.013891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:51.013891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:51.013891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:51.014678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:51.014678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:51.014678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:51.014678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:51.014678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:51.014678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:51.014678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:51.014678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:51.014678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:51.014678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:51.014678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:06:51.014678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:06:51.015436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 196/386] 10.222.0.26 () {56 vars in 1114 bytes} [Tue Sep 27 04:06:50 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:06:51.023482 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f940f5bb-d8ea-449f-aaaa-4c39b86e9026 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:51.050004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d599bae4-1001-42d8-a2a1-499a05a37817 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:51.050616 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d599bae4-1001-42d8-a2a1-499a05a37817 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:51.050898 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d599bae4-1001-42d8-a2a1-499a05a37817 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:51.051168 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d599bae4-1001-42d8-a2a1-499a05a37817 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:51.052408 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d599bae4-1001-42d8-a2a1-499a05a37817 None admin] RBAC: Adding query filter params (name=member) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:51.052751 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d599bae4-1001-42d8-a2a1-499a05a37817 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:51.055012 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d599bae4-1001-42d8-a2a1-499a05a37817 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:51.071366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 191/387] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:50 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:51.077146 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d599bae4-1001-42d8-a2a1-499a05a37817 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:51.081093 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:51.128112 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:51.210932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f940f5bb-d8ea-449f-aaaa-4c39b86e9026 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:51.250895 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a745b89-7372-4db5-b994-ca42cca5089d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:51.251507 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a745b89-7372-4db5-b994-ca42cca5089d None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:51.251789 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a745b89-7372-4db5-b994-ca42cca5089d None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:51.252061 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a745b89-7372-4db5-b994-ca42cca5089d None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:51.252988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a745b89-7372-4db5-b994-ca42cca5089d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:51.253288 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a745b89-7372-4db5-b994-ca42cca5089d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:51.255162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a745b89-7372-4db5-b994-ca42cca5089d None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:51.261425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d599bae4-1001-42d8-a2a1-499a05a37817 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:51.267011 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 197/388] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:06:51 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:51.272731 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a745b89-7372-4db5-b994-ca42cca5089d None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:51.324738 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1e7fe9-2721-442e-a535-9bb90c32b3bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:51.325364 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1e7fe9-2721-442e-a535-9bb90c32b3bc None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:51.325637 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1e7fe9-2721-442e-a535-9bb90c32b3bc None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:51.325930 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1e7fe9-2721-442e-a535-9bb90c32b3bc None admin] PATH_INFO: `/v3/groups/1b59b23587524103b43265b13a5157d1` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:51.327641 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1e7fe9-2721-442e-a535-9bb90c32b3bc None admin] RBAC: Authorizing `identity:get_group(group_id=1b59b23587524103b43265b13a5157d1)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:51.329064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1e7fe9-2721-442e-a535-9bb90c32b3bc None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:51.331704 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 192/389] 10.222.0.26 () {56 vars in 1180 bytes} [Tue Sep 27 04:06:51 2022] GET /identity/v3/groups/1b59b23587524103b43265b13a5157d1 => generated 226 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:51.340575 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1e7fe9-2721-442e-a535-9bb90c32b3bc None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:51.430115 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:51.453793 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a745b89-7372-4db5-b994-ca42cca5089d None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:51.507637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b63046-90f6-4871-b1eb-58bfb5d5ccdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:51.508277 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b63046-90f6-4871-b1eb-58bfb5d5ccdd None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:51.508558 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b63046-90f6-4871-b1eb-58bfb5d5ccdd None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:51.508828 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b63046-90f6-4871-b1eb-58bfb5d5ccdd None admin] PATH_INFO: `/v3/groups/1b59b23587524103b43265b13a5157d1` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:51.510587 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b63046-90f6-4871-b1eb-58bfb5d5ccdd None admin] RBAC: Authorizing `identity:get_group(group_id=1b59b23587524103b43265b13a5157d1)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:51.512542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b63046-90f6-4871-b1eb-58bfb5d5ccdd None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:51.514551 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 198/390] 10.222.0.26 () {56 vars in 1180 bytes} [Tue Sep 27 04:06:51 2022] GET /identity/v3/groups/1b59b23587524103b43265b13a5157d1 => generated 226 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:51.520307 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b63046-90f6-4871-b1eb-58bfb5d5ccdd None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:51.523309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:51.535319 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1e7fe9-2721-442e-a535-9bb90c32b3bc None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:51.600492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eca619d-2ea9-4ff7-931d-01cce684a116 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:51.601475 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eca619d-2ea9-4ff7-931d-01cce684a116 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:51.601945 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eca619d-2ea9-4ff7-931d-01cce684a116 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:51.602376 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eca619d-2ea9-4ff7-931d-01cce684a116 None admin] PATH_INFO: `/v3/projects/146fd1189c56413c9d17cd16d1a974d3` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:51.603922 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eca619d-2ea9-4ff7-931d-01cce684a116 None admin] RBAC: Authorizing `identity:get_project(project_id=146fd1189c56413c9d17cd16d1a974d3)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:51.605526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eca619d-2ea9-4ff7-931d-01cce684a116 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:51.611339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 193/391] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:51 2022] GET /identity/v3/projects/146fd1189c56413c9d17cd16d1a974d3 => generated 298 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:51.617629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eca619d-2ea9-4ff7-931d-01cce684a116 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:51.654343 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b63046-90f6-4871-b1eb-58bfb5d5ccdd None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:51.695307 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db86c2c-90c9-4a16-bc43-0fc7b35619eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:51.695964 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db86c2c-90c9-4a16-bc43-0fc7b35619eb None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:51.696242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db86c2c-90c9-4a16-bc43-0fc7b35619eb None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:51.696512 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db86c2c-90c9-4a16-bc43-0fc7b35619eb None admin] PATH_INFO: `/v3/projects/146fd1189c56413c9d17cd16d1a974d3` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:51.697904 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db86c2c-90c9-4a16-bc43-0fc7b35619eb None admin] RBAC: Authorizing `identity:get_project(project_id=146fd1189c56413c9d17cd16d1a974d3)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:51.699887 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db86c2c-90c9-4a16-bc43-0fc7b35619eb None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:51.701716 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 199/392] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:51 2022] GET /identity/v3/projects/146fd1189c56413c9d17cd16d1a974d3 => generated 298 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:51.707596 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db86c2c-90c9-4a16-bc43-0fc7b35619eb None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:51.759805 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:51.785685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eca619d-2ea9-4ff7-931d-01cce684a116 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:51.823295 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4a1623-9c41-42cc-893e-55987406b9a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:51.823954 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4a1623-9c41-42cc-893e-55987406b9a1 None admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:51.824226 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4a1623-9c41-42cc-893e-55987406b9a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:51.824493 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4a1623-9c41-42cc-893e-55987406b9a1 None admin] PATH_INFO: `/v3/projects/146fd1189c56413c9d17cd16d1a974d3/groups/1b59b23587524103b43265b13a5157d1/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:51.828162 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4a1623-9c41-42cc-893e-55987406b9a1 None admin] RBAC: Authorizing `identity:create_grant(project_id=146fd1189c56413c9d17cd16d1a974d3, group_id=1b59b23587524103b43265b13a5157d1, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:51.829599 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4a1623-9c41-42cc-893e-55987406b9a1 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:51.851666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 194/393] 10.222.0.26 () {58 vars in 1442 bytes} [Tue Sep 27 04:06:51 2022] PUT /identity/v3/projects/146fd1189c56413c9d17cd16d1a974d3/groups/1b59b23587524103b43265b13a5157d1/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:06:51.913376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db86c2c-90c9-4a16-bc43-0fc7b35619eb None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:51.956629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86411fc9-3415-4be4-b6b4-761439f1d333 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:51.957425 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86411fc9-3415-4be4-b6b4-761439f1d333 None admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:51.957710 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86411fc9-3415-4be4-b6b4-761439f1d333 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:51.958063 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86411fc9-3415-4be4-b6b4-761439f1d333 None admin] PATH_INFO: `/v3/projects/146fd1189c56413c9d17cd16d1a974d3/groups/1b59b23587524103b43265b13a5157d1/roles/2d9f391485604d0b952b7c844c3a3266` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:51.961114 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86411fc9-3415-4be4-b6b4-761439f1d333 None admin] RBAC: Authorizing `identity:create_grant(project_id=146fd1189c56413c9d17cd16d1a974d3, group_id=1b59b23587524103b43265b13a5157d1, role_id=2d9f391485604d0b952b7c844c3a3266)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:51.962754 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86411fc9-3415-4be4-b6b4-761439f1d333 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:51.981720 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 200/394] 10.222.0.26 () {58 vars in 1442 bytes} [Tue Sep 27 04:06:51 2022] PUT /identity/v3/projects/146fd1189c56413c9d17cd16d1a974d3/groups/1b59b23587524103b43265b13a5157d1/roles/2d9f391485604d0b952b7c844c3a3266 => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:06:52.097220 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b74d1e-93a4-4b73-b00f-258e1fb1c0ce None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:52.097764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b74d1e-93a4-4b73-b00f-258e1fb1c0ce None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:52.097764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b74d1e-93a4-4b73-b00f-258e1fb1c0ce None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:52.099514 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 195/395] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:52.105274 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f321cc1-8676-4a35-848a-2b5aa7cb3c05 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:52.105555 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f321cc1-8676-4a35-848a-2b5aa7cb3c05 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:52.105852 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f321cc1-8676-4a35-848a-2b5aa7cb3c05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:52.126486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-6f321cc1-8676-4a35-848a-2b5aa7cb3c05 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:52.127984 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-6f321cc1-8676-4a35-848a-2b5aa7cb3c05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:52.222064 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 201/396] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:52.510485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b74d1e-93a4-4b73-b00f-258e1fb1c0ce None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:52.538997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:52.644367 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b74d1e-93a4-4b73-b00f-258e1fb1c0ce None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:52.687777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc9dfe3-939c-4d49-ac4d-6fa4cc6823ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:52.688330 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc9dfe3-939c-4d49-ac4d-6fa4cc6823ef None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:52.688918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc9dfe3-939c-4d49-ac4d-6fa4cc6823ef None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:52.689189 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc9dfe3-939c-4d49-ac4d-6fa4cc6823ef None admin] PATH_INFO: `/v3/roles/member` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:52.701780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc9dfe3-939c-4d49-ac4d-6fa4cc6823ef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:52.707172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-fcc9dfe3-939c-4d49-ac4d-6fa4cc6823ef None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:52.707172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:52.707172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:52.707172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:52.707172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:52.707172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:52.707172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:52.707172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:52.707172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:52.707172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:52.707172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:52.707172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:52.707172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:52.707172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:52.707172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:52.707172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:52.708137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:52.708137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:52.708137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:52.708137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:52.708137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:52.708137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:52.708137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:52.708137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:52.708137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:52.708137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:52.708137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:52.708137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:52.708137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:52.708137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:52.708137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:52.708137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:52.709355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:52.709355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:52.709355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:52.709355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:52.709355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:52.709355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:52.709355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:52.709355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:52.709355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:52.709355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:52.709355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:52.709355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:06:52.709355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 196/397] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:52 2022] GET /identity/v3/roles/member => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:52.714504 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f321cc1-8676-4a35-848a-2b5aa7cb3c05 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:52.729972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:52.759477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb81aef-5c77-4fc8-a7d2-d94cf0c148a4 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:52.759797 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb81aef-5c77-4fc8-a7d2-d94cf0c148a4 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:52.760014 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb81aef-5c77-4fc8-a7d2-d94cf0c148a4 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:52.761484 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 197/398] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:06:52.769680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ff3fc2-feca-439c-85bc-984c15ad5870 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:52.769983 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ff3fc2-feca-439c-85bc-984c15ad5870 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:52.770237 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ff3fc2-feca-439c-85bc-984c15ad5870 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:52.796120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-01ff3fc2-feca-439c-85bc-984c15ad5870 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:52.797305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-01ff3fc2-feca-439c-85bc-984c15ad5870 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:52.861575 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f321cc1-8676-4a35-848a-2b5aa7cb3c05 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:52.891450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 198/399] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:06:52.903490 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec902cc2-5a9f-4b9e-9984-a5e193863f2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:52.904249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec902cc2-5a9f-4b9e-9984-a5e193863f2c None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:52.904632 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec902cc2-5a9f-4b9e-9984-a5e193863f2c None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:52.905006 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec902cc2-5a9f-4b9e-9984-a5e193863f2c None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:52.905801 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec902cc2-5a9f-4b9e-9984-a5e193863f2c None admin] RBAC: Adding query filter params (name=member) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:52.906206 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec902cc2-5a9f-4b9e-9984-a5e193863f2c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:52.907659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec902cc2-5a9f-4b9e-9984-a5e193863f2c None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:52.917329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 202/400] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:52 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:52.922542 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ff3fc2-feca-439c-85bc-984c15ad5870 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:52.947628 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:53.063792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ff3fc2-feca-439c-85bc-984c15ad5870 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:53.110111 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa4a8b6-2e22-41bb-8424-617ae7d8522a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:53.110731 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa4a8b6-2e22-41bb-8424-617ae7d8522a None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:53.111005 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa4a8b6-2e22-41bb-8424-617ae7d8522a None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:53.111264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa4a8b6-2e22-41bb-8424-617ae7d8522a None admin] PATH_INFO: `/v3/groups/1b59b23587524103b43265b13a5157d1` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:53.112373 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa4a8b6-2e22-41bb-8424-617ae7d8522a None admin] RBAC: Authorizing `identity:get_group(group_id=1b59b23587524103b43265b13a5157d1)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:53.113813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa4a8b6-2e22-41bb-8424-617ae7d8522a None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:53.115698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 199/401] 10.222.0.26 () {56 vars in 1180 bytes} [Tue Sep 27 04:06:52 2022] GET /identity/v3/groups/1b59b23587524103b43265b13a5157d1 => generated 226 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:53.121156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec902cc2-5a9f-4b9e-9984-a5e193863f2c None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:53.214154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:53.235064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa4a8b6-2e22-41bb-8424-617ae7d8522a None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:53.254138 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec902cc2-5a9f-4b9e-9984-a5e193863f2c None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:53.300801 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e245dd06-9cd2-4d1a-ad6e-5b54f8c14f55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:53.301847 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e245dd06-9cd2-4d1a-ad6e-5b54f8c14f55 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:53.302133 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e245dd06-9cd2-4d1a-ad6e-5b54f8c14f55 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:53.302408 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e245dd06-9cd2-4d1a-ad6e-5b54f8c14f55 None admin] PATH_INFO: `/v3/projects/bbe6cda6a9564434b77be066e776bd55` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:53.306456 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e245dd06-9cd2-4d1a-ad6e-5b54f8c14f55 None admin] RBAC: Authorizing `identity:get_project(project_id=bbe6cda6a9564434b77be066e776bd55)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:53.307899 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e245dd06-9cd2-4d1a-ad6e-5b54f8c14f55 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:53.309924 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 203/402] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:53 2022] GET /identity/v3/projects/bbe6cda6a9564434b77be066e776bd55 => generated 302 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:53.319884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e245dd06-9cd2-4d1a-ad6e-5b54f8c14f55 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:53.359573 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:53.377489 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa4a8b6-2e22-41bb-8424-617ae7d8522a None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:53.433217 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d19e85c-44aa-42aa-9fd7-66acd4400f8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:53.434054 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d19e85c-44aa-42aa-9fd7-66acd4400f8f None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:53.434329 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d19e85c-44aa-42aa-9fd7-66acd4400f8f None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:53.435119 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d19e85c-44aa-42aa-9fd7-66acd4400f8f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:53.447440 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d19e85c-44aa-42aa-9fd7-66acd4400f8f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:53.453126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-0d19e85c-44aa-42aa-9fd7-66acd4400f8f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:06:53.453126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:53.453126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:53.453126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:53.453126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:53.453126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:53.453126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:53.453126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:53.453126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:53.453126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:53.453126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:53.453126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:53.453126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:53.453126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:53.453126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:53.453126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:53.454055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:53.454055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:53.454055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:53.454055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:53.454055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:53.454055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:53.454055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:53.454055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:53.454055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:53.454055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:53.454055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:53.454055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:53.454055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:53.454055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:53.454055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:53.454055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:53.454865 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:53.454865 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:53.454865 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:53.454865 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:53.454865 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:53.454865 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:53.454865 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:53.454865 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:53.454865 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:53.454865 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:53.454865 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:06:53.454865 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:06:53.454865 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 200/403] 10.222.0.26 () {56 vars in 1114 bytes} [Tue Sep 27 04:06:53 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:06:53.462192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d19e85c-44aa-42aa-9fd7-66acd4400f8f None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:53.563346 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e245dd06-9cd2-4d1a-ad6e-5b54f8c14f55 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:53.607171 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d77afb-6b39-4caa-a982-9014e3a5ab32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:53.607844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d77afb-6b39-4caa-a982-9014e3a5ab32 None admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:53.608016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d19e85c-44aa-42aa-9fd7-66acd4400f8f None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:53.608177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d77afb-6b39-4caa-a982-9014e3a5ab32 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:53.608388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d77afb-6b39-4caa-a982-9014e3a5ab32 None admin] PATH_INFO: `/v3/projects/bbe6cda6a9564434b77be066e776bd55/groups/1b59b23587524103b43265b13a5157d1/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:53.610505 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d77afb-6b39-4caa-a982-9014e3a5ab32 None admin] RBAC: Authorizing `identity:create_grant(project_id=bbe6cda6a9564434b77be066e776bd55, group_id=1b59b23587524103b43265b13a5157d1, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:53.611939 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d77afb-6b39-4caa-a982-9014e3a5ab32 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:53.634221 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 204/404] 10.222.0.26 () {58 vars in 1442 bytes} [Tue Sep 27 04:06:53 2022] PUT /identity/v3/projects/bbe6cda6a9564434b77be066e776bd55/groups/1b59b23587524103b43265b13a5157d1/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:06:53.635608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beac10be-e5a9-4b37-aa62-01e5c1b7095c None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:53.635947 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beac10be-e5a9-4b37-aa62-01e5c1b7095c None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:53.636286 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beac10be-e5a9-4b37-aa62-01e5c1b7095c None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:53.637914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 205/405] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:53.643835 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c752d1dc-47b7-4dfd-9afa-9c87256466a6 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:53.644166 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c752d1dc-47b7-4dfd-9afa-9c87256466a6 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:53.644452 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c752d1dc-47b7-4dfd-9afa-9c87256466a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:53.660396 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:53.673607 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b42521-cf59-4bec-8214-7d77f37998da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:53.674313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b42521-cf59-4bec-8214-7d77f37998da None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:53.674699 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b42521-cf59-4bec-8214-7d77f37998da None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:53.674952 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b42521-cf59-4bec-8214-7d77f37998da None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:53.676603 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b42521-cf59-4bec-8214-7d77f37998da None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:53.678343 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b42521-cf59-4bec-8214-7d77f37998da None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:53.681502 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b42521-cf59-4bec-8214-7d77f37998da None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:53.683922 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-c752d1dc-47b7-4dfd-9afa-9c87256466a6 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:53.687836 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c752d1dc-47b7-4dfd-9afa-9c87256466a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:53.703490 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 201/406] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:06:53 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:53.708802 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b42521-cf59-4bec-8214-7d77f37998da None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:53.715246 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:53.841853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b42521-cf59-4bec-8214-7d77f37998da None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:53.862677 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 206/407] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:06:53.885331 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb6dcae-3acf-4487-988b-9d5111b80cf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:53.886333 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb6dcae-3acf-4487-988b-9d5111b80cf4 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:53.886840 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb6dcae-3acf-4487-988b-9d5111b80cf4 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:53.887318 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb6dcae-3acf-4487-988b-9d5111b80cf4 None admin] PATH_INFO: `/v3/groups/1b59b23587524103b43265b13a5157d1` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:53.888615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb6dcae-3acf-4487-988b-9d5111b80cf4 None admin] RBAC: Authorizing `identity:get_group(group_id=1b59b23587524103b43265b13a5157d1)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:53.890424 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb6dcae-3acf-4487-988b-9d5111b80cf4 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:53.892560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 202/408] 10.222.0.26 () {56 vars in 1180 bytes} [Tue Sep 27 04:06:53 2022] GET /identity/v3/groups/1b59b23587524103b43265b13a5157d1 => generated 226 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:53.897793 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c752d1dc-47b7-4dfd-9afa-9c87256466a6 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:54.039865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c752d1dc-47b7-4dfd-9afa-9c87256466a6 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:54.084730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:54.089467 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7214fa-c43a-4157-87ee-85e19bf1d56e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:54.090218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7214fa-c43a-4157-87ee-85e19bf1d56e None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:54.090498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7214fa-c43a-4157-87ee-85e19bf1d56e None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:54.090833 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7214fa-c43a-4157-87ee-85e19bf1d56e None admin] PATH_INFO: `/v3/projects/bbe6cda6a9564434b77be066e776bd55` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:54.092027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7214fa-c43a-4157-87ee-85e19bf1d56e None admin] RBAC: Authorizing `identity:get_project(project_id=bbe6cda6a9564434b77be066e776bd55)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:54.093659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7214fa-c43a-4157-87ee-85e19bf1d56e None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:54.100600 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 207/409] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:53 2022] GET /identity/v3/projects/bbe6cda6a9564434b77be066e776bd55 => generated 302 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:54.101488 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb6dcae-3acf-4487-988b-9d5111b80cf4 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:54.192594 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:54.231373 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7214fa-c43a-4157-87ee-85e19bf1d56e None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:54.237055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb6dcae-3acf-4487-988b-9d5111b80cf4 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:54.298454 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9b2f88-aa46-46da-9276-c78f977e8533 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:54.299131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9b2f88-aa46-46da-9276-c78f977e8533 None admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:54.299410 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9b2f88-aa46-46da-9276-c78f977e8533 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:54.299681 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9b2f88-aa46-46da-9276-c78f977e8533 None admin] PATH_INFO: `/v3/projects/bbe6cda6a9564434b77be066e776bd55/groups/1b59b23587524103b43265b13a5157d1/roles/2d9f391485604d0b952b7c844c3a3266` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:54.304548 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9b2f88-aa46-46da-9276-c78f977e8533 None admin] RBAC: Authorizing `identity:create_grant(project_id=bbe6cda6a9564434b77be066e776bd55, group_id=1b59b23587524103b43265b13a5157d1, role_id=2d9f391485604d0b952b7c844c3a3266)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:54.306011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9b2f88-aa46-46da-9276-c78f977e8533 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:54.332472 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 203/410] 10.222.0.26 () {58 vars in 1442 bytes} [Tue Sep 27 04:06:54 2022] PUT /identity/v3/projects/bbe6cda6a9564434b77be066e776bd55/groups/1b59b23587524103b43265b13a5157d1/roles/2d9f391485604d0b952b7c844c3a3266 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:06:54.365100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7214fa-c43a-4157-87ee-85e19bf1d56e None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:54.403358 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e836e982-5b1d-43b6-bdc5-13497fa39110 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:54.404009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e836e982-5b1d-43b6-bdc5-13497fa39110 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:54.404282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e836e982-5b1d-43b6-bdc5-13497fa39110 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:54.404556 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e836e982-5b1d-43b6-bdc5-13497fa39110 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:54.418124 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e836e982-5b1d-43b6-bdc5-13497fa39110 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:54.423107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-e836e982-5b1d-43b6-bdc5-13497fa39110 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:06:54.423107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:54.423107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:54.423107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:54.423107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:54.423107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:54.423107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:54.423107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:54.423107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:54.423107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:54.423107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:54.423107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:54.423107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:54.423107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:54.423107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:54.423107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:54.424312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:54.424312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:54.424312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:54.424312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:54.424312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:54.424312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:54.424312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:54.424312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:54.424312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:54.424312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:54.424312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:54.424312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:54.424312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:54.424312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:54.424312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:54.424312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:54.425149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:54.425149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:54.425149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:54.425149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:54.425149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:54.425149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:54.425149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:54.425149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:54.425149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:54.425149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:54.425149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:06:54.425149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:06:54.425149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 208/411] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:06:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:06:54.428853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9b2f88-aa46-46da-9276-c78f977e8533 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:54.594568 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9b2f88-aa46-46da-9276-c78f977e8533 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:54.631737 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:54.637717 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d831d238-7f4b-4f25-9be5-a46faf9360f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:54.638354 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d831d238-7f4b-4f25-9be5-a46faf9360f5 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:54.639322 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d831d238-7f4b-4f25-9be5-a46faf9360f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:54.639602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d831d238-7f4b-4f25-9be5-a46faf9360f5 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:54.640274 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d831d238-7f4b-4f25-9be5-a46faf9360f5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:54.641224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d831d238-7f4b-4f25-9be5-a46faf9360f5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:54.642704 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d831d238-7f4b-4f25-9be5-a46faf9360f5 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:54.656583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 204/412] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:54.662504 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e836e982-5b1d-43b6-bdc5-13497fa39110 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:54.732233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:54.782914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e836e982-5b1d-43b6-bdc5-13497fa39110 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:54.817963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec7790c-bba9-4a49-b15b-280a1adcb6bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:54.818680 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec7790c-bba9-4a49-b15b-280a1adcb6bd None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:54.818965 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec7790c-bba9-4a49-b15b-280a1adcb6bd None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:54.819239 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec7790c-bba9-4a49-b15b-280a1adcb6bd None admin] PATH_INFO: `/v3/groups/4564949965a9439ca5dae605a38a0650` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:54.820372 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec7790c-bba9-4a49-b15b-280a1adcb6bd None admin] RBAC: Authorizing `identity:get_group(group_id=4564949965a9439ca5dae605a38a0650)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:54.821839 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec7790c-bba9-4a49-b15b-280a1adcb6bd None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:54.823763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 209/413] 10.222.0.26 () {56 vars in 1180 bytes} [Tue Sep 27 04:06:54 2022] GET /identity/v3/groups/4564949965a9439ca5dae605a38a0650 => generated 229 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:54.827912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3a4ad7-99a2-482a-8539-eeab7ac62057 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:54.828625 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3a4ad7-99a2-482a-8539-eeab7ac62057 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:54.829056 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec7790c-bba9-4a49-b15b-280a1adcb6bd None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:54.829301 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3a4ad7-99a2-482a-8539-eeab7ac62057 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:54.832174 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 205/414] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:54.837043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb49189-f5c6-4eeb-8b5a-587a66963261 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:54.837561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb49189-f5c6-4eeb-8b5a-587a66963261 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:54.838073 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb49189-f5c6-4eeb-8b5a-587a66963261 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:54.857031 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-feb49189-f5c6-4eeb-8b5a-587a66963261 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:54.858622 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-feb49189-f5c6-4eeb-8b5a-587a66963261 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:54.947894 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec7790c-bba9-4a49-b15b-280a1adcb6bd None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:54.986183 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:54.991968 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c1be3e-700a-4b89-a02b-f1338a496a1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:54.992989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c1be3e-700a-4b89-a02b-f1338a496a1b None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:54.993504 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c1be3e-700a-4b89-a02b-f1338a496a1b None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:54.995186 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c1be3e-700a-4b89-a02b-f1338a496a1b None admin] PATH_INFO: `/v3/projects/3cc5c8e99cb64893a8d035d6aa4f3eb6` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:54.999634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c1be3e-700a-4b89-a02b-f1338a496a1b None admin] RBAC: Authorizing `identity:get_project(project_id=3cc5c8e99cb64893a8d035d6aa4f3eb6)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:55.002342 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c1be3e-700a-4b89-a02b-f1338a496a1b None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:55.008482 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 210/415] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:54 2022] GET /identity/v3/projects/3cc5c8e99cb64893a8d035d6aa4f3eb6 => generated 344 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:55.013339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c1be3e-700a-4b89-a02b-f1338a496a1b None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:55.033778 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 206/416] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:55.107291 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec113b7-3a8d-4a9b-9f19-7c4e94141c0f None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:55.107609 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec113b7-3a8d-4a9b-9f19-7c4e94141c0f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:55.107840 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec113b7-3a8d-4a9b-9f19-7c4e94141c0f None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:55.109170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 207/417] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:06:55.114775 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b9b0f7-6102-41ee-b6b4-19e0b590d361 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:55.115294 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b9b0f7-6102-41ee-b6b4-19e0b590d361 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:55.115571 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b9b0f7-6102-41ee-b6b4-19e0b590d361 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:55.137612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-14b9b0f7-6102-41ee-b6b4-19e0b590d361 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:55.139616 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-14b9b0f7-6102-41ee-b6b4-19e0b590d361 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:55.204567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c1be3e-700a-4b89-a02b-f1338a496a1b None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:55.230676 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 208/418] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:55.295319 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97914d3d-0fb3-4ac2-89b0-7c9bbb66f236 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:55.295976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97914d3d-0fb3-4ac2-89b0-7c9bbb66f236 None admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:55.296254 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97914d3d-0fb3-4ac2-89b0-7c9bbb66f236 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:55.296526 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97914d3d-0fb3-4ac2-89b0-7c9bbb66f236 None admin] PATH_INFO: `/v3/projects/3cc5c8e99cb64893a8d035d6aa4f3eb6/groups/4564949965a9439ca5dae605a38a0650/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:55.303491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97914d3d-0fb3-4ac2-89b0-7c9bbb66f236 None admin] RBAC: Authorizing `identity:create_grant(project_id=3cc5c8e99cb64893a8d035d6aa4f3eb6, group_id=4564949965a9439ca5dae605a38a0650, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:55.304927 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97914d3d-0fb3-4ac2-89b0-7c9bbb66f236 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:55.329239 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 211/419] 10.222.0.26 () {58 vars in 1442 bytes} [Tue Sep 27 04:06:55 2022] PUT /identity/v3/projects/3cc5c8e99cb64893a8d035d6aa4f3eb6/groups/4564949965a9439ca5dae605a38a0650/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:06:55.358435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b9b0f7-6102-41ee-b6b4-19e0b590d361 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:55.484991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:55.563035 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b9b0f7-6102-41ee-b6b4-19e0b590d361 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:55.588951 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97914d3d-0fb3-4ac2-89b0-7c9bbb66f236 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:55.604564 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0a30f3-f60c-4980-953c-2a500d8d09d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:55.605686 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0a30f3-f60c-4980-953c-2a500d8d09d7 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:55.606197 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0a30f3-f60c-4980-953c-2a500d8d09d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:55.606659 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0a30f3-f60c-4980-953c-2a500d8d09d7 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:55.618653 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0a30f3-f60c-4980-953c-2a500d8d09d7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:55.624526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-2a0a30f3-f60c-4980-953c-2a500d8d09d7 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:06:55.624526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:55.624526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:55.624526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:55.624526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:55.624526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:55.624526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:55.624526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:55.624526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:55.624526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:55.624526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:55.624526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:55.624526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:55.624526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:55.624526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:55.624526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:55.625410 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:55.625410 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:55.625410 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:55.625410 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:55.625410 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:55.625410 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:55.625410 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:55.625410 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:55.625410 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:55.625410 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:55.625410 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:55.625410 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:55.625410 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:55.625410 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:55.625410 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:55.625410 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:55.626232 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:55.626232 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:55.626232 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:55.626232 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:55.626232 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:55.626232 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:55.626232 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:55.626232 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:55.626232 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:55.626232 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:55.626232 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:06:55.626232 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:06:55.628817 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 209/420] 10.222.0.26 () {56 vars in 1114 bytes} [Tue Sep 27 04:06:55 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:55.636950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0a30f3-f60c-4980-953c-2a500d8d09d7 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:55.654470 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:55.709158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97914d3d-0fb3-4ac2-89b0-7c9bbb66f236 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:55.748395 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28429355-4357-43ad-8f98-b6f136dd15ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:55.748946 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28429355-4357-43ad-8f98-b6f136dd15ee None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:55.749227 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28429355-4357-43ad-8f98-b6f136dd15ee None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:55.749502 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28429355-4357-43ad-8f98-b6f136dd15ee None admin] PATH_INFO: `/v3/roles/member` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:55.756906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0a30f3-f60c-4980-953c-2a500d8d09d7 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:55.761125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28429355-4357-43ad-8f98-b6f136dd15ee None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:55.766226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-28429355-4357-43ad-8f98-b6f136dd15ee None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:55.766226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:55.766226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:55.766226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:55.766226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:55.766226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:55.766226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:55.766226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:55.766226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:55.766226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:55.766226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:55.766226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:55.766226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:55.766226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:55.766226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:55.766226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:55.767104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:55.767104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:55.767104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:55.767104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:55.767104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:55.767104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:55.767104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:55.767104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:55.767104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:55.767104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:55.767104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:55.767104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:55.767104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:55.767104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:55.767104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:55.767104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:55.768356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:55.768356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:55.768356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:55.768356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:55.768356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:55.768356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:55.768356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:55.768356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:55.768356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:55.768356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:55.768356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:55.768356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:06:55.768356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 212/421] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:55 2022] GET /identity/v3/roles/member => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:06:55.775755 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:55.779241 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28429355-4357-43ad-8f98-b6f136dd15ee None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:55.800356 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4d479d-7b07-4e87-b810-b1f79b08bbab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:55.801349 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4d479d-7b07-4e87-b810-b1f79b08bbab None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:55.802519 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4d479d-7b07-4e87-b810-b1f79b08bbab None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:55.803055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4d479d-7b07-4e87-b810-b1f79b08bbab None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:55.804039 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4d479d-7b07-4e87-b810-b1f79b08bbab None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:55.804536 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4d479d-7b07-4e87-b810-b1f79b08bbab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:55.806252 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4d479d-7b07-4e87-b810-b1f79b08bbab None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:55.820360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 210/422] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:06:55 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:55.821927 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4d479d-7b07-4e87-b810-b1f79b08bbab None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:55.938364 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28429355-4357-43ad-8f98-b6f136dd15ee None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:55.972067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:55.993165 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4d479d-7b07-4e87-b810-b1f79b08bbab None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:56.018659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25192198-4b31-4341-bb8c-7db340fdb905 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:56.019205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25192198-4b31-4341-bb8c-7db340fdb905 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:56.019489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25192198-4b31-4341-bb8c-7db340fdb905 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:56.019760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25192198-4b31-4341-bb8c-7db340fdb905 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:56.020460 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25192198-4b31-4341-bb8c-7db340fdb905 None admin] RBAC: Adding query filter params (name=member) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:56.021308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25192198-4b31-4341-bb8c-7db340fdb905 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:56.022739 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25192198-4b31-4341-bb8c-7db340fdb905 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:56.040414 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1e3c83c-ff01-4c66-9d9f-2d0348aa79ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:56.041394 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e3c83c-ff01-4c66-9d9f-2d0348aa79ca None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:56.041922 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 213/423] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:55 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:56.042180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e3c83c-ff01-4c66-9d9f-2d0348aa79ca None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:56.042703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e3c83c-ff01-4c66-9d9f-2d0348aa79ca None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:56.047190 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25192198-4b31-4341-bb8c-7db340fdb905 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:56.096615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:56.136926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e3c83c-ff01-4c66-9d9f-2d0348aa79ca None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:56.138925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e3c83c-ff01-4c66-9d9f-2d0348aa79ca None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:56.205775 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25192198-4b31-4341-bb8c-7db340fdb905 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:56.247102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a42e1d-81f0-4b37-94cf-375c015ded0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:56.248142 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a42e1d-81f0-4b37-94cf-375c015ded0c None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:56.248435 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a42e1d-81f0-4b37-94cf-375c015ded0c None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:56.248709 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a42e1d-81f0-4b37-94cf-375c015ded0c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:56.265950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 211/424] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 439 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:06:56.268290 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba71145-46cd-4af4-a0c2-38c8a836a2d9 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:56.268798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba71145-46cd-4af4-a0c2-38c8a836a2d9 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:56.269224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba71145-46cd-4af4-a0c2-38c8a836a2d9 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:56.273059 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 212/425] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:56.274373 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba71145-46cd-4af4-a0c2-38c8a836a2d9 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:56.306237 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:56.331960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a42e1d-81f0-4b37-94cf-375c015ded0c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:56.334219 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a42e1d-81f0-4b37-94cf-375c015ded0c None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:56.363927 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:56.390857 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba71145-46cd-4af4-a0c2-38c8a836a2d9 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:56.441361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207ab342-e44b-4daa-a6b2-de2be394bec3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:56.442309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207ab342-e44b-4daa-a6b2-de2be394bec3 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:56.443263 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207ab342-e44b-4daa-a6b2-de2be394bec3 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:56.443735 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207ab342-e44b-4daa-a6b2-de2be394bec3 None admin] PATH_INFO: `/v3/projects/146fd1189c56413c9d17cd16d1a974d3` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:56.445083 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-207ab342-e44b-4daa-a6b2-de2be394bec3 None admin] RBAC: Authorizing `identity:get_project(project_id=146fd1189c56413c9d17cd16d1a974d3)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:56.446846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-207ab342-e44b-4daa-a6b2-de2be394bec3 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:56.448759 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 213/426] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:56 2022] GET /identity/v3/projects/146fd1189c56413c9d17cd16d1a974d3 => generated 298 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:56.454491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4febf27-15a9-4ad1-8650-195f8877b493 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:56.455466 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4febf27-15a9-4ad1-8650-195f8877b493 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:56.455919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4febf27-15a9-4ad1-8650-195f8877b493 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:56.457847 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 214/427] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 27 04:06:56.459321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a42e1d-81f0-4b37-94cf-375c015ded0c None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:56.476580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-d4febf27-15a9-4ad1-8650-195f8877b493 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:56.478255 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d4febf27-15a9-4ad1-8650-195f8877b493 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:56.550972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:56.555409 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 214/428] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:56.557440 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4febf27-15a9-4ad1-8650-195f8877b493 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:56.597238 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a42e1d-81f0-4b37-94cf-375c015ded0c None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:56.660615 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f58fe21-2bc6-4b5c-a6dc-5bb752f97754 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:56.661542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f58fe21-2bc6-4b5c-a6dc-5bb752f97754 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:56.662003 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f58fe21-2bc6-4b5c-a6dc-5bb752f97754 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:56.662433 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f58fe21-2bc6-4b5c-a6dc-5bb752f97754 None admin] PATH_INFO: `/v3/groups/1b59b23587524103b43265b13a5157d1` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:56.664447 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f58fe21-2bc6-4b5c-a6dc-5bb752f97754 None admin] RBAC: Authorizing `identity:get_group(group_id=1b59b23587524103b43265b13a5157d1)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:56.666639 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f58fe21-2bc6-4b5c-a6dc-5bb752f97754 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:56.673770 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 215/429] 10.222.0.26 () {56 vars in 1180 bytes} [Tue Sep 27 04:06:56 2022] GET /identity/v3/groups/1b59b23587524103b43265b13a5157d1 => generated 226 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:56.675154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f58fe21-2bc6-4b5c-a6dc-5bb752f97754 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:56.696055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:56.747230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4febf27-15a9-4ad1-8650-195f8877b493 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:56.815596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bfc5f0-f9af-429a-bd88-f4784727dfe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:56.818785 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bfc5f0-f9af-429a-bd88-f4784727dfe3 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:56.819299 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bfc5f0-f9af-429a-bd88-f4784727dfe3 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:56.819714 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bfc5f0-f9af-429a-bd88-f4784727dfe3 None admin] PATH_INFO: `/v3/projects/146fd1189c56413c9d17cd16d1a974d3` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:56.821314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06bfc5f0-f9af-429a-bd88-f4784727dfe3 None admin] RBAC: Authorizing `identity:get_project(project_id=146fd1189c56413c9d17cd16d1a974d3)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:56.823170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06bfc5f0-f9af-429a-bd88-f4784727dfe3 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:56.825246 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 215/430] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:56 2022] GET /identity/v3/projects/146fd1189c56413c9d17cd16d1a974d3 => generated 298 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:56.830695 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bfc5f0-f9af-429a-bd88-f4784727dfe3 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:56.876422 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f58fe21-2bc6-4b5c-a6dc-5bb752f97754 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:56.916775 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66398c5a-7d9d-44af-a6d8-07a1c5094703 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:56.917402 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66398c5a-7d9d-44af-a6d8-07a1c5094703 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:56.917682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66398c5a-7d9d-44af-a6d8-07a1c5094703 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:56.917985 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66398c5a-7d9d-44af-a6d8-07a1c5094703 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:56.919249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66398c5a-7d9d-44af-a6d8-07a1c5094703 None admin] RBAC: Adding query filter params (group.id=1b59b23587524103b43265b13a5157d1, role.id=2d9f391485604d0b952b7c844c3a3266, scope.project.id=146fd1189c56413c9d17cd16d1a974d3) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:56.919249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66398c5a-7d9d-44af-a6d8-07a1c5094703 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:56.920492 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66398c5a-7d9d-44af-a6d8-07a1c5094703 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:56.933907 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 216/431] 10.222.0.26 () {56 vars in 1376 bytes} [Tue Sep 27 04:06:56 2022] GET /identity/v3/role_assignments?group.id=1b59b23587524103b43265b13a5157d1&scope.project.id=146fd1189c56413c9d17cd16d1a974d3&role.id=2d9f391485604d0b952b7c844c3a3266 => generated 611 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:56.933907 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66398c5a-7d9d-44af-a6d8-07a1c5094703 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:56.956370 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:56.969947 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bfc5f0-f9af-429a-bd88-f4784727dfe3 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:57.008495 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e152eb-2900-4d25-8a9d-66bb72693970 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:57.009480 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e152eb-2900-4d25-8a9d-66bb72693970 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:57.010019 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e152eb-2900-4d25-8a9d-66bb72693970 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:57.010569 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e152eb-2900-4d25-8a9d-66bb72693970 None admin] PATH_INFO: `/v3/groups/1b59b23587524103b43265b13a5157d1` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:57.011900 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e152eb-2900-4d25-8a9d-66bb72693970 None admin] RBAC: Authorizing `identity:get_group(group_id=1b59b23587524103b43265b13a5157d1)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:57.013594 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e152eb-2900-4d25-8a9d-66bb72693970 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:57.015869 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 216/432] 10.222.0.26 () {56 vars in 1180 bytes} [Tue Sep 27 04:06:56 2022] GET /identity/v3/groups/1b59b23587524103b43265b13a5157d1 => generated 226 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:57.022725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e152eb-2900-4d25-8a9d-66bb72693970 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:57.040799 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:57.094124 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66398c5a-7d9d-44af-a6d8-07a1c5094703 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:57.133182 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7f4295-bab5-437c-ad5c-441cb04a9234 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:57.133956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7f4295-bab5-437c-ad5c-441cb04a9234 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:57.134242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7f4295-bab5-437c-ad5c-441cb04a9234 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:57.134520 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7f4295-bab5-437c-ad5c-441cb04a9234 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:57.141615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e152eb-2900-4d25-8a9d-66bb72693970 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:57.146791 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7f4295-bab5-437c-ad5c-441cb04a9234 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:57.152631 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-8f7f4295-bab5-437c-ad5c-441cb04a9234 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:57.152631 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:57.152631 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:57.152631 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:57.152631 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:57.152631 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:57.152631 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:57.152631 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:57.152631 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:57.152631 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:57.152631 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:57.152631 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:57.152631 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:57.152631 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:57.152631 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:57.152631 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:57.153764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:57.153764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:57.153764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:57.153764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:57.153764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:57.153764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:57.153764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:57.153764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:57.153764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:57.153764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:57.153764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:57.153764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:57.153764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:57.153764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:57.153764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:57.153764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:57.154634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:57.154634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:57.154634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:57.154634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:57.154634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:57.154634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:57.154634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:57.154634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:57.154634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:57.154634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:57.154634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 04:06:57.154634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:06:57.158681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 217/433] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:56 2022] GET /identity/v3/roles/member => generated 84 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:57.160211 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f685f3-34ab-4e41-b44a-b130cbe7d118 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:57.160501 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f685f3-34ab-4e41-b44a-b130cbe7d118 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:57.160817 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f685f3-34ab-4e41-b44a-b130cbe7d118 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:57.167407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 218/434] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:57.168337 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f685f3-34ab-4e41-b44a-b130cbe7d118 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:57.189336 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46766713-f240-4f77-b50d-480807c901b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:57.189970 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46766713-f240-4f77-b50d-480807c901b5 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:57.190247 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46766713-f240-4f77-b50d-480807c901b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:57.190520 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46766713-f240-4f77-b50d-480807c901b5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:57.191285 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46766713-f240-4f77-b50d-480807c901b5 None admin] RBAC: Adding query filter params (group.id=1b59b23587524103b43265b13a5157d1, role.id=e3ce5a3697ec4292a446366d3caadeea, scope.project.id=146fd1189c56413c9d17cd16d1a974d3) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:57.191588 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46766713-f240-4f77-b50d-480807c901b5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:57.193129 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46766713-f240-4f77-b50d-480807c901b5 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:57.199933 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 217/435] 10.222.0.26 () {56 vars in 1376 bytes} [Tue Sep 27 04:06:57 2022] GET /identity/v3/role_assignments?group.id=1b59b23587524103b43265b13a5157d1&scope.project.id=146fd1189c56413c9d17cd16d1a974d3&role.id=e3ce5a3697ec4292a446366d3caadeea => generated 611 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:57.201089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889e610f-294d-4537-b6da-768dcf911a66 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:57.201387 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889e610f-294d-4537-b6da-768dcf911a66 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:57.201641 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889e610f-294d-4537-b6da-768dcf911a66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:57.234817 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-889e610f-294d-4537-b6da-768dcf911a66 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:57.236649 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-889e610f-294d-4537-b6da-768dcf911a66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:57.249073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:57.290919 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f685f3-34ab-4e41-b44a-b130cbe7d118 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:57.321720 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 218/436] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:57.334859 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902d1a2b-114b-4c3f-9c83-e8e9ac386085 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:57.335885 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902d1a2b-114b-4c3f-9c83-e8e9ac386085 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:57.336401 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902d1a2b-114b-4c3f-9c83-e8e9ac386085 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:57.336880 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902d1a2b-114b-4c3f-9c83-e8e9ac386085 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:57.337794 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902d1a2b-114b-4c3f-9c83-e8e9ac386085 None admin] RBAC: Adding query filter params (name=member) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:57.338403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902d1a2b-114b-4c3f-9c83-e8e9ac386085 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:57.340058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902d1a2b-114b-4c3f-9c83-e8e9ac386085 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:57.351635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 219/437] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:06:57 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:57.357348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889e610f-294d-4537-b6da-768dcf911a66 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:57.362080 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:57.469712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889e610f-294d-4537-b6da-768dcf911a66 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:57.505669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c33b840-c854-46af-9b77-3183cf55ab94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:57.506357 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c33b840-c854-46af-9b77-3183cf55ab94 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:57.506643 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c33b840-c854-46af-9b77-3183cf55ab94 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:57.506914 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c33b840-c854-46af-9b77-3183cf55ab94 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:57.578463 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c33b840-c854-46af-9b77-3183cf55ab94 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:57.579922 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c33b840-c854-46af-9b77-3183cf55ab94 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:57.584002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:57.619849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902d1a2b-114b-4c3f-9c83-e8e9ac386085 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:57.707409 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 219/438] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 27 04:06:57.714444 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c33b840-c854-46af-9b77-3183cf55ab94 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:57.740121 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902d1a2b-114b-4c3f-9c83-e8e9ac386085 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:57.787779 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:57.793457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa96275-13fe-4f5d-b7e0-e91c3a75a1aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:57.794467 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa96275-13fe-4f5d-b7e0-e91c3a75a1aa None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:57.796281 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa96275-13fe-4f5d-b7e0-e91c3a75a1aa None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:57.796281 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa96275-13fe-4f5d-b7e0-e91c3a75a1aa None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:57.806585 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa96275-13fe-4f5d-b7e0-e91c3a75a1aa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:57.811547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-1fa96275-13fe-4f5d-b7e0-e91c3a75a1aa None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:06:57.811547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:57.811547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:57.811547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:57.811547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:57.811547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:57.811547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:57.811547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:57.811547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:57.811547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:57.811547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:57.811547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:57.811547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:57.811547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:57.811547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:57.811547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:57.812424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:57.812424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:57.812424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:57.812424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:57.812424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:57.812424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:57.812424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:57.812424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:57.812424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:57.812424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:57.812424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:57.812424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:57.812424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:57.812424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:57.812424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:57.812424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:57.813925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:57.813925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:57.813925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:57.813925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:57.813925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:57.813925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:57.813925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:57.813925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:57.813925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:57.813925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:57.813925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:06:57.813925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:06:57.813925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 220/439] 10.222.0.26 () {56 vars in 1114 bytes} [Tue Sep 27 04:06:57 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:57.815353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8e6a7c-77f6-4f75-98cb-48c3977243e7 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:57.815637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8e6a7c-77f6-4f75-98cb-48c3977243e7 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:57.815909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8e6a7c-77f6-4f75-98cb-48c3977243e7 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:57.817187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 221/440] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:06:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:06:57.819587 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8e6a7c-77f6-4f75-98cb-48c3977243e7 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:57.840454 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c33b840-c854-46af-9b77-3183cf55ab94 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:57.843160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:57.882852 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662b7171-b3e9-4019-9968-0db39d40785b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:57.883518 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662b7171-b3e9-4019-9968-0db39d40785b None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:57.883802 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662b7171-b3e9-4019-9968-0db39d40785b None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:57.884074 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662b7171-b3e9-4019-9968-0db39d40785b None admin] PATH_INFO: `/v3/projects/bbe6cda6a9564434b77be066e776bd55` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:57.885452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662b7171-b3e9-4019-9968-0db39d40785b None admin] RBAC: Authorizing `identity:get_project(project_id=bbe6cda6a9564434b77be066e776bd55)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:57.886661 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662b7171-b3e9-4019-9968-0db39d40785b None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:57.890966 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 220/441] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:57 2022] GET /identity/v3/projects/bbe6cda6a9564434b77be066e776bd55 => generated 302 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:57.892249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74b0f6b-f4dc-4d62-b272-059909b98679 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:57.892249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74b0f6b-f4dc-4d62-b272-059909b98679 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:57.892433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74b0f6b-f4dc-4d62-b272-059909b98679 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:57.913595 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-e74b0f6b-f4dc-4d62-b272-059909b98679 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:06:57.914899 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-e74b0f6b-f4dc-4d62-b272-059909b98679 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:06:57.946679 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8e6a7c-77f6-4f75-98cb-48c3977243e7 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:57.980066 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a370cd2d-6826-4dcc-8c84-15e506ed09ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:57.980655 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a370cd2d-6826-4dcc-8c84-15e506ed09ed None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:57.980933 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a370cd2d-6826-4dcc-8c84-15e506ed09ed None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:57.981205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a370cd2d-6826-4dcc-8c84-15e506ed09ed None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:57.981891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a370cd2d-6826-4dcc-8c84-15e506ed09ed None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:57.982191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a370cd2d-6826-4dcc-8c84-15e506ed09ed None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:57.983535 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a370cd2d-6826-4dcc-8c84-15e506ed09ed None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:57.984081 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 221/442] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:06:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:06:57.984904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74b0f6b-f4dc-4d62-b272-059909b98679 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:57.991440 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 222/443] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:06:57 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:57.996914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a370cd2d-6826-4dcc-8c84-15e506ed09ed None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:58.055681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:58.098953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74b0f6b-f4dc-4d62-b272-059909b98679 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:58.100706 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:58.102135 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a370cd2d-6826-4dcc-8c84-15e506ed09ed None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:58.135497 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc845902-64bf-478e-a09e-29c4dafbeaea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:58.136346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc845902-64bf-478e-a09e-29c4dafbeaea None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:58.136644 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc845902-64bf-478e-a09e-29c4dafbeaea None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:58.136894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc845902-64bf-478e-a09e-29c4dafbeaea None admin] PATH_INFO: `/v3/groups/1b59b23587524103b43265b13a5157d1` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:58.137713 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7193e1f9-6a5e-4f60-a487-e1c77557cb8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:58.138012 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc845902-64bf-478e-a09e-29c4dafbeaea None admin] RBAC: Authorizing `identity:get_group(group_id=1b59b23587524103b43265b13a5157d1)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:58.138330 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7193e1f9-6a5e-4f60-a487-e1c77557cb8e None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:58.138608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7193e1f9-6a5e-4f60-a487-e1c77557cb8e None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:58.138878 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7193e1f9-6a5e-4f60-a487-e1c77557cb8e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:58.139433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc845902-64bf-478e-a09e-29c4dafbeaea None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:58.141321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 222/444] 10.222.0.26 () {56 vars in 1180 bytes} [Tue Sep 27 04:06:57 2022] GET /identity/v3/groups/1b59b23587524103b43265b13a5157d1 => generated 226 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:58.146665 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc845902-64bf-478e-a09e-29c4dafbeaea None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:58.208796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7193e1f9-6a5e-4f60-a487-e1c77557cb8e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:58.210277 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7193e1f9-6a5e-4f60-a487-e1c77557cb8e None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:58.249448 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc845902-64bf-478e-a09e-29c4dafbeaea None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:58.267817 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:58.283423 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed21b913-c0fc-4188-9099-cd6699ddf6fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:58.284010 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed21b913-c0fc-4188-9099-cd6699ddf6fd None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:58.284281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed21b913-c0fc-4188-9099-cd6699ddf6fd None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:58.284547 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed21b913-c0fc-4188-9099-cd6699ddf6fd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:58.285262 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed21b913-c0fc-4188-9099-cd6699ddf6fd None admin] RBAC: Adding query filter params (group.id=1b59b23587524103b43265b13a5157d1, role.id=e3ce5a3697ec4292a446366d3caadeea, scope.project.id=bbe6cda6a9564434b77be066e776bd55) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:58.285565 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed21b913-c0fc-4188-9099-cd6699ddf6fd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:58.287040 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed21b913-c0fc-4188-9099-cd6699ddf6fd None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:58.292734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 223/445] 10.222.0.26 () {56 vars in 1376 bytes} [Tue Sep 27 04:06:58 2022] GET /identity/v3/role_assignments?group.id=1b59b23587524103b43265b13a5157d1&scope.project.id=bbe6cda6a9564434b77be066e776bd55&role.id=e3ce5a3697ec4292a446366d3caadeea => generated 611 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:58.293282 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed21b913-c0fc-4188-9099-cd6699ddf6fd None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:58.325348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 223/446] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 27 04:06:58.331399 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7193e1f9-6a5e-4f60-a487-e1c77557cb8e None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:58.332314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:58.404992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed21b913-c0fc-4188-9099-cd6699ddf6fd None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:58.447468 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7193e1f9-6a5e-4f60-a487-e1c77557cb8e None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:58.451266 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5298fdb7-a70e-40f7-a4d4-48f2f41d3d87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:58.451912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5298fdb7-a70e-40f7-a4d4-48f2f41d3d87 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:58.452190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5298fdb7-a70e-40f7-a4d4-48f2f41d3d87 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:58.452461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5298fdb7-a70e-40f7-a4d4-48f2f41d3d87 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:58.463912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5298fdb7-a70e-40f7-a4d4-48f2f41d3d87 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:06:58.469321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-5298fdb7-a70e-40f7-a4d4-48f2f41d3d87 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:06:58.469321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:06:58.469321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:06:58.469321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:06:58.469321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:06:58.469321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:06:58.469321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:06:58.469321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:06:58.469321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:06:58.469321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:06:58.469321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:06:58.469321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:06:58.469321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:06:58.469321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:06:58.469321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:06:58.469321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:06:58.470225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:06:58.470225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:06:58.470225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:06:58.470225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:06:58.470225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:06:58.470225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:06:58.470225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:06:58.470225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:06:58.470225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:06:58.470225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:06:58.470225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:06:58.470225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:06:58.470225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:06:58.470225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:06:58.470225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:06:58.470225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:06:58.471011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:06:58.471011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:06:58.471011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:06:58.471011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:06:58.471011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:06:58.471011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:06:58.471011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:06:58.471011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:06:58.471011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:06:58.471011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:06:58.471011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:06:58.471011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:06:58.471011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 224/447] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:06:58 2022] GET /identity/v3/roles/admin => generated 83 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:06:58.476835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5298fdb7-a70e-40f7-a4d4-48f2f41d3d87 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:58.488023 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b37b56-dd3b-4f1e-8952-f3af5b70d397 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:58.488657 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b37b56-dd3b-4f1e-8952-f3af5b70d397 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:58.488936 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b37b56-dd3b-4f1e-8952-f3af5b70d397 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:58.489215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b37b56-dd3b-4f1e-8952-f3af5b70d397 None admin] PATH_INFO: `/v3/projects/bbe6cda6a9564434b77be066e776bd55` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:58.490935 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b37b56-dd3b-4f1e-8952-f3af5b70d397 None admin] RBAC: Authorizing `identity:get_project(project_id=bbe6cda6a9564434b77be066e776bd55)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:58.492365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b37b56-dd3b-4f1e-8952-f3af5b70d397 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:58.497815 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 224/448] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:58 2022] GET /identity/v3/projects/bbe6cda6a9564434b77be066e776bd55 => generated 302 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:58.500030 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b37b56-dd3b-4f1e-8952-f3af5b70d397 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:58.527530 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:58.593507 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:58.600046 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5298fdb7-a70e-40f7-a4d4-48f2f41d3d87 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:58.620498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b37b56-dd3b-4f1e-8952-f3af5b70d397 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:58.643401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b26ddc-f943-4f4d-b244-11cffcdd04ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:58.644060 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b26ddc-f943-4f4d-b244-11cffcdd04ea None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:58.644340 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b26ddc-f943-4f4d-b244-11cffcdd04ea None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:58.644609 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b26ddc-f943-4f4d-b244-11cffcdd04ea None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:58.645273 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b26ddc-f943-4f4d-b244-11cffcdd04ea None admin] RBAC: Adding query filter params (name=admin) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:58.645605 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b26ddc-f943-4f4d-b244-11cffcdd04ea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:58.647039 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b26ddc-f943-4f4d-b244-11cffcdd04ea None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:58.656204 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 225/449] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:06:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:58.660325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5fae4ff-10e6-4638-a460-bccd0a1a91ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:58.661085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fae4ff-10e6-4638-a460-bccd0a1a91ee None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:58.662054 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fae4ff-10e6-4638-a460-bccd0a1a91ee None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:58.662329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fae4ff-10e6-4638-a460-bccd0a1a91ee None admin] PATH_INFO: `/v3/groups/1b59b23587524103b43265b13a5157d1` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:58.662519 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b26ddc-f943-4f4d-b244-11cffcdd04ea None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:58.663467 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5fae4ff-10e6-4638-a460-bccd0a1a91ee None admin] RBAC: Authorizing `identity:get_group(group_id=1b59b23587524103b43265b13a5157d1)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:58.664888 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5fae4ff-10e6-4638-a460-bccd0a1a91ee None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:58.666726 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 225/450] 10.222.0.26 () {56 vars in 1180 bytes} [Tue Sep 27 04:06:58 2022] GET /identity/v3/groups/1b59b23587524103b43265b13a5157d1 => generated 226 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:58.673323 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5fae4ff-10e6-4638-a460-bccd0a1a91ee None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:58.763448 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:58.763647 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b26ddc-f943-4f4d-b244-11cffcdd04ea None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:58.773021 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5fae4ff-10e6-4638-a460-bccd0a1a91ee None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:58.796849 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95053c6-7f2c-4f51-9b10-601d2b2a9ab4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:58.797437 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95053c6-7f2c-4f51-9b10-601d2b2a9ab4 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:58.797711 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95053c6-7f2c-4f51-9b10-601d2b2a9ab4 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:58.798010 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95053c6-7f2c-4f51-9b10-601d2b2a9ab4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:58.807507 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b240b4-ab3d-4cbc-9d22-d1cabdebd0cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:58.808096 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b240b4-ab3d-4cbc-9d22-d1cabdebd0cf None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:58.808379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b240b4-ab3d-4cbc-9d22-d1cabdebd0cf None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:58.808649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b240b4-ab3d-4cbc-9d22-d1cabdebd0cf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:58.809368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b240b4-ab3d-4cbc-9d22-d1cabdebd0cf None admin] RBAC: Adding query filter params (group.id=1b59b23587524103b43265b13a5157d1, role.id=2d9f391485604d0b952b7c844c3a3266, scope.project.id=bbe6cda6a9564434b77be066e776bd55) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:58.809680 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b240b4-ab3d-4cbc-9d22-d1cabdebd0cf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:58.811119 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b240b4-ab3d-4cbc-9d22-d1cabdebd0cf None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:58.816805 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 226/451] 10.222.0.26 () {56 vars in 1376 bytes} [Tue Sep 27 04:06:58 2022] GET /identity/v3/role_assignments?group.id=1b59b23587524103b43265b13a5157d1&scope.project.id=bbe6cda6a9564434b77be066e776bd55&role.id=2d9f391485604d0b952b7c844c3a3266 => generated 611 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:06:58.823213 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:58.869538 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95053c6-7f2c-4f51-9b10-601d2b2a9ab4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:58.871056 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95053c6-7f2c-4f51-9b10-601d2b2a9ab4 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:58.978731 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 226/452] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:06:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 27 04:06:58.980886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b240b4-ab3d-4cbc-9d22-d1cabdebd0cf None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:59.083825 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b240b4-ab3d-4cbc-9d22-d1cabdebd0cf None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:59.119118 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8ba9a8-fb80-49cd-b3a9-df1c0543fdae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:59.119745 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8ba9a8-fb80-49cd-b3a9-df1c0543fdae None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:59.120025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8ba9a8-fb80-49cd-b3a9-df1c0543fdae None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:59.120298 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8ba9a8-fb80-49cd-b3a9-df1c0543fdae None admin] PATH_INFO: `/v3/projects/3cc5c8e99cb64893a8d035d6aa4f3eb6` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:59.121412 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8ba9a8-fb80-49cd-b3a9-df1c0543fdae None admin] RBAC: Authorizing `identity:get_project(project_id=3cc5c8e99cb64893a8d035d6aa4f3eb6)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:59.122908 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8ba9a8-fb80-49cd-b3a9-df1c0543fdae None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:59.124665 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 227/453] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:06:58 2022] GET /identity/v3/projects/3cc5c8e99cb64893a8d035d6aa4f3eb6 => generated 344 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:59.130433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95053c6-7f2c-4f51-9b10-601d2b2a9ab4 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:59.186403 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:59.228878 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95053c6-7f2c-4f51-9b10-601d2b2a9ab4 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:59.261411 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986922fc-becb-4a73-bee0-15b8f352f4bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:59.262061 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986922fc-becb-4a73-bee0-15b8f352f4bf None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:59.262339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986922fc-becb-4a73-bee0-15b8f352f4bf None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:59.262611 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986922fc-becb-4a73-bee0-15b8f352f4bf None admin] PATH_INFO: `/v3/groups/4564949965a9439ca5dae605a38a0650` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:59.263696 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986922fc-becb-4a73-bee0-15b8f352f4bf None admin] RBAC: Authorizing `identity:get_group(group_id=4564949965a9439ca5dae605a38a0650)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:59.265133 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986922fc-becb-4a73-bee0-15b8f352f4bf None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:59.266720 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 227/454] 10.222.0.26 () {56 vars in 1180 bytes} [Tue Sep 27 04:06:59 2022] GET /identity/v3/groups/4564949965a9439ca5dae605a38a0650 => generated 229 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:06:59.273333 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8ba9a8-fb80-49cd-b3a9-df1c0543fdae None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:06:59.297143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:06:59.368484 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8ba9a8-fb80-49cd-b3a9-df1c0543fdae None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:06:59.400749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8796706c-3844-4ac4-810d-e3121c101e79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:06:59.401357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8796706c-3844-4ac4-810d-e3121c101e79 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:06:59.401633 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8796706c-3844-4ac4-810d-e3121c101e79 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:06:59.401929 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8796706c-3844-4ac4-810d-e3121c101e79 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:06:59.402644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8796706c-3844-4ac4-810d-e3121c101e79 None admin] RBAC: Adding query filter params (group.id=4564949965a9439ca5dae605a38a0650, role.id=e6a31562a18649f0809202c36a8e18ef, scope.project.id=3cc5c8e99cb64893a8d035d6aa4f3eb6) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:06:59.402947 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8796706c-3844-4ac4-810d-e3121c101e79 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:06:59.404359 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8796706c-3844-4ac4-810d-e3121c101e79 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:06:59.410117 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 228/455] 10.222.0.26 () {56 vars in 1376 bytes} [Tue Sep 27 04:06:59 2022] GET /identity/v3/role_assignments?group.id=4564949965a9439ca5dae605a38a0650&scope.project.id=3cc5c8e99cb64893a8d035d6aa4f3eb6&role.id=e6a31562a18649f0809202c36a8e18ef => generated 611 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:01.968349 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3305e7eb-e63c-4834-81ef-143022378bb6 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:01.969188 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3305e7eb-e63c-4834-81ef-143022378bb6 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:01.969598 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3305e7eb-e63c-4834-81ef-143022378bb6 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:01.971348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 228/456] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:01.977712 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f98341c-d29f-4551-8bc9-47aba2a15efe None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:01.978589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f98341c-d29f-4551-8bc9-47aba2a15efe None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:01.979052 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f98341c-d29f-4551-8bc9-47aba2a15efe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:02.006693 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-2f98341c-d29f-4551-8bc9-47aba2a15efe None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:02.008209 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-2f98341c-d29f-4551-8bc9-47aba2a15efe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:02.099282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 229/457] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:07:02.321324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dcd872-ff1d-4eb2-b5bb-c3dde640aa31 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:02.321324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dcd872-ff1d-4eb2-b5bb-c3dde640aa31 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:02.321324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dcd872-ff1d-4eb2-b5bb-c3dde640aa31 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:02.321939 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 229/458] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:02.327952 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cccc5e-23d1-4d4d-bf19-ae9f33804fd4 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:02.328359 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cccc5e-23d1-4d4d-bf19-ae9f33804fd4 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:02.328728 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cccc5e-23d1-4d4d-bf19-ae9f33804fd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:02.351700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-66cccc5e-23d1-4d4d-bf19-ae9f33804fd4 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:02.352889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-66cccc5e-23d1-4d4d-bf19-ae9f33804fd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:02.368272 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:02.441198 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20dcd872-ff1d-4eb2-b5bb-c3dde640aa31 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:02.444304 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 230/459] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:07:02.579865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc86e9a1-134a-424e-bb87-cdac3f228972 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:02.580489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc86e9a1-134a-424e-bb87-cdac3f228972 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:02.580983 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc86e9a1-134a-424e-bb87-cdac3f228972 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:02.582797 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 231/460] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:07:02.589321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecae537-fd08-490e-b113-9d9e17e4aa20 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:02.589879 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecae537-fd08-490e-b113-9d9e17e4aa20 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:02.590350 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecae537-fd08-490e-b113-9d9e17e4aa20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:02.620467 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-8ecae537-fd08-490e-b113-9d9e17e4aa20 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:02.621661 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-8ecae537-fd08-490e-b113-9d9e17e4aa20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:02.634523 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20dcd872-ff1d-4eb2-b5bb-c3dde640aa31 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:02.671525 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:02.692446 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46128c3e-cc8c-4edd-93cb-487ec2776119 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:02.693081 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46128c3e-cc8c-4edd-93cb-487ec2776119 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:02.693354 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46128c3e-cc8c-4edd-93cb-487ec2776119 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:02.693625 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46128c3e-cc8c-4edd-93cb-487ec2776119 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:02.710191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46128c3e-cc8c-4edd-93cb-487ec2776119 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:02.711638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46128c3e-cc8c-4edd-93cb-487ec2776119 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:02.733992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-46128c3e-cc8c-4edd-93cb-487ec2776119 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:02.733992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:02.733992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:02.733992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:02.733992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:02.733992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:02.733992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:02.733992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:02.733992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:02.733992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:02.733992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:02.733992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:02.733992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:02.733992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:02.733992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:02.733992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:02.734949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:02.734949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:02.734949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:02.734949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:02.734949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:02.734949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:02.734949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:02.734949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:02.734949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:02.734949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:02.734949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:02.734949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:02.734949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:02.734949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:02.734949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:02.734949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:02.734949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:02.735831 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:02.735831 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:02.735831 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:02.735831 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 230/461] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:07:02.747427 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46128c3e-cc8c-4edd-93cb-487ec2776119 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:02.752236 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 232/462] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:07:02.762575 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ecae537-fd08-490e-b113-9d9e17e4aa20 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:02.914185 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:02.936580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46128c3e-cc8c-4edd-93cb-487ec2776119 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:02.976802 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ecae537-fd08-490e-b113-9d9e17e4aa20 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:02.980273 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926df5a5-3324-4294-8ca1-977e170b2aaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:02.981461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926df5a5-3324-4294-8ca1-977e170b2aaf None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:02.981958 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926df5a5-3324-4294-8ca1-977e170b2aaf None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:02.982427 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926df5a5-3324-4294-8ca1-977e170b2aaf None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:02.983263 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926df5a5-3324-4294-8ca1-977e170b2aaf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:02.983757 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926df5a5-3324-4294-8ca1-977e170b2aaf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:02.985772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926df5a5-3324-4294-8ca1-977e170b2aaf None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:03.009874 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 231/463] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:03.011346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ee2aa5-2675-4185-981c-cb3167ceff96 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:03.011824 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ee2aa5-2675-4185-981c-cb3167ceff96 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:03.012290 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ee2aa5-2675-4185-981c-cb3167ceff96 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:03.013681 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 232/464] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:03.018679 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ee2aa5-2675-4185-981c-cb3167ceff96 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:03.021391 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b86296-1bad-4eb2-947e-2db6fce3ef3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:03.022179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b86296-1bad-4eb2-947e-2db6fce3ef3a None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:03.022523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b86296-1bad-4eb2-947e-2db6fce3ef3a None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:03.022909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b86296-1bad-4eb2-947e-2db6fce3ef3a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:03.036322 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b86296-1bad-4eb2-947e-2db6fce3ef3a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:03.038299 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b86296-1bad-4eb2-947e-2db6fce3ef3a None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:03.058760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-f0b86296-1bad-4eb2-947e-2db6fce3ef3a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:03.058760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:03.058760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:03.058760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:03.058760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:03.058760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:03.058760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:03.058760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:03.058760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:03.058760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:03.058760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:03.058760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:03.058760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:03.058760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:03.058760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:03.058760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:03.059683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:03.059683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:03.059683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:03.059683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:03.059683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:03.059683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:03.059683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:03.059683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:03.059683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:03.059683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:03.059683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:03.059683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:03.059683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:03.059683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:03.059683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:03.059683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:03.059683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:03.060525 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:03.060525 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:03.060525 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:03.060525 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 233/465] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 298 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:03.066384 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8aee47-615f-4e3d-9bc0-789d58137330 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:03.066739 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8aee47-615f-4e3d-9bc0-789d58137330 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:03.068728 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:03.068859 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8aee47-615f-4e3d-9bc0-789d58137330 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:03.088622 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-3c8aee47-615f-4e3d-9bc0-789d58137330 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:03.089984 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-3c8aee47-615f-4e3d-9bc0-789d58137330 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:03.140873 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ee2aa5-2675-4185-981c-cb3167ceff96 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:03.161190 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 234/466] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:07:03.162308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8aee47-615f-4e3d-9bc0-789d58137330 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:03.178726 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ced4286-618b-4e2d-9543-727c51eee7e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:03.179732 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ced4286-618b-4e2d-9543-727c51eee7e2 None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:03.180247 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ced4286-618b-4e2d-9543-727c51eee7e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:03.180754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ced4286-618b-4e2d-9543-727c51eee7e2 None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:03.181717 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ced4286-618b-4e2d-9543-727c51eee7e2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:03.183702 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ced4286-618b-4e2d-9543-727c51eee7e2 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:03.210551 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 233/467] 10.222.0.26 () {60 vars in 1132 bytes} [Tue Sep 27 04:07:03 2022] POST /identity/v3/users => generated 246 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:07:03.212409 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ced4286-618b-4e2d-9543-727c51eee7e2 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:03.233326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:03.336729 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8aee47-615f-4e3d-9bc0-789d58137330 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:03.346811 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:03.358957 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ced4286-618b-4e2d-9543-727c51eee7e2 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:03.409274 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4867d39-8de0-4606-a563-8a8307a05a71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:03.409959 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4867d39-8de0-4606-a563-8a8307a05a71 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:03.410239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4867d39-8de0-4606-a563-8a8307a05a71 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:03.410509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4867d39-8de0-4606-a563-8a8307a05a71 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:03.421380 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961c7024-d5b7-4052-a6fb-cb3966aa3d79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:03.422360 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961c7024-d5b7-4052-a6fb-cb3966aa3d79 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:03.422835 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961c7024-d5b7-4052-a6fb-cb3966aa3d79 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:03.423869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961c7024-d5b7-4052-a6fb-cb3966aa3d79 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:03.425000 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961c7024-d5b7-4052-a6fb-cb3966aa3d79 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:03.425490 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961c7024-d5b7-4052-a6fb-cb3966aa3d79 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:03.428282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961c7024-d5b7-4052-a6fb-cb3966aa3d79 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:03.431462 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4867d39-8de0-4606-a563-8a8307a05a71 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:03.432856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4867d39-8de0-4606-a563-8a8307a05a71 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:03.444796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 235/468] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:03.448049 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961c7024-d5b7-4052-a6fb-cb3966aa3d79 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:03.457130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-b4867d39-8de0-4606-a563-8a8307a05a71 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:03.457130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:03.457130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:03.457130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:03.457130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:03.457130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:03.457130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:03.457130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:03.457130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:03.457130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:03.457130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:03.457130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:03.457130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:03.457130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:03.457130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:03.457130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:03.458406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:03.458406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:03.458406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:03.458406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:03.458406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:03.458406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:03.458406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:03.458406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:03.458406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:03.458406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:03.458406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:03.458406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:03.458406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:03.458406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:03.458406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:03.458406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:03.458406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:03.459649 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:03.459649 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:03.459649 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:03.459649 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 234/469] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:03.467004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4867d39-8de0-4606-a563-8a8307a05a71 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:03.569651 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961c7024-d5b7-4052-a6fb-cb3966aa3d79 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:03.589260 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4867d39-8de0-4606-a563-8a8307a05a71 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:03.598158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:03.606563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bbbc9d-45b5-4513-9c13-b9bcec312742 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:03.607191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bbbc9d-45b5-4513-9c13-b9bcec312742 None admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:03.607462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bbbc9d-45b5-4513-9c13-b9bcec312742 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:03.608199 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bbbc9d-45b5-4513-9c13-b9bcec312742 None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:03.608781 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bbbc9d-45b5-4513-9c13-b9bcec312742 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:03.610505 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bbbc9d-45b5-4513-9c13-b9bcec312742 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:03.636164 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed261e8-1806-4419-b486-e778d8c0b9a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:03.637790 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 236/470] 10.222.0.26 () {60 vars in 1132 bytes} [Tue Sep 27 04:07:03 2022] POST /identity/v3/users => generated 248 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:07:03.638377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bbbc9d-45b5-4513-9c13-b9bcec312742 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:03.642068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed261e8-1806-4419-b486-e778d8c0b9a2 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:03.643300 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed261e8-1806-4419-b486-e778d8c0b9a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:03.643593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed261e8-1806-4419-b486-e778d8c0b9a2 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:03.645651 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed261e8-1806-4419-b486-e778d8c0b9a2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:03.646618 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed261e8-1806-4419-b486-e778d8c0b9a2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:03.650087 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed261e8-1806-4419-b486-e778d8c0b9a2 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:03.677879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 235/471] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:03.683215 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed261e8-1806-4419-b486-e778d8c0b9a2 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:03.689522 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:03.851569 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bbbc9d-45b5-4513-9c13-b9bcec312742 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:03.859875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed261e8-1806-4419-b486-e778d8c0b9a2 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:03.893138 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a87ed97-6740-44dc-86ab-e4753460fac5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:03.893791 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a87ed97-6740-44dc-86ab-e4753460fac5 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:03.894074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a87ed97-6740-44dc-86ab-e4753460fac5 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:03.894343 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a87ed97-6740-44dc-86ab-e4753460fac5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:03.901613 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3a400c-24b5-4e00-bac3-f5552e6d0633 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:03.902255 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3a400c-24b5-4e00-bac3-f5552e6d0633 None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:03.902531 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3a400c-24b5-4e00-bac3-f5552e6d0633 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:03.902803 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3a400c-24b5-4e00-bac3-f5552e6d0633 None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:03.903451 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3a400c-24b5-4e00-bac3-f5552e6d0633 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:03.904880 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3a400c-24b5-4e00-bac3-f5552e6d0633 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:03.906156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a87ed97-6740-44dc-86ab-e4753460fac5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:03.907761 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a87ed97-6740-44dc-86ab-e4753460fac5 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:03.928315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-3a87ed97-6740-44dc-86ab-e4753460fac5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:03.928315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:03.928315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:03.928315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:03.928315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:03.928315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:03.928315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:03.928315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:03.928315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:03.928315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:03.928315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:03.928315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:03.928315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:03.928315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:03.928315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:03.928315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:03.929255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:03.929255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:03.929255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:03.929255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:03.929255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:03.929255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:03.929255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:03.929255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:03.929255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:03.929255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:03.929255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:03.929255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:03.929255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:03.929255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:03.929255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:03.929255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:03.929255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:03.930115 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:03.930115 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:03.930115 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:03.931567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 237/472] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:03.932888 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3409a225-1cc8-42b0-9dc5-d6b44ea1c1e7 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:03.933424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3409a225-1cc8-42b0-9dc5-d6b44ea1c1e7 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:03.933869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3409a225-1cc8-42b0-9dc5-d6b44ea1c1e7 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:03.935447 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 238/473] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:03.938797 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3409a225-1cc8-42b0-9dc5-d6b44ea1c1e7 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:03.941071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 236/474] 10.222.0.26 () {60 vars in 1132 bytes} [Tue Sep 27 04:07:03 2022] POST /identity/v3/users => generated 248 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:07:03.942286 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1057d4-8cb3-4462-9eec-0262bd41ec0f None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:03.942603 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1057d4-8cb3-4462-9eec-0262bd41ec0f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:03.942879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1057d4-8cb3-4462-9eec-0262bd41ec0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:03.964039 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-db1057d4-8cb3-4462-9eec-0262bd41ec0f None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:03.965457 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-db1057d4-8cb3-4462-9eec-0262bd41ec0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:03.996492 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:04.044952 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:04.071315 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 237/475] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:07:04.134848 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3409a225-1cc8-42b0-9dc5-d6b44ea1c1e7 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:04.183233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3cc3f1-bcff-4c64-9ca2-fc7ae65cdde9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:04.183895 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3cc3f1-bcff-4c64-9ca2-fc7ae65cdde9 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:04.184118 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3cc3f1-bcff-4c64-9ca2-fc7ae65cdde9 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:04.184390 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3cc3f1-bcff-4c64-9ca2-fc7ae65cdde9 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:04.185036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3cc3f1-bcff-4c64-9ca2-fc7ae65cdde9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:04.185332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3cc3f1-bcff-4c64-9ca2-fc7ae65cdde9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:04.186722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3cc3f1-bcff-4c64-9ca2-fc7ae65cdde9 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:04.199626 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 239/476] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:04.205104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1057d4-8cb3-4462-9eec-0262bd41ec0f None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:04.336811 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1057d4-8cb3-4462-9eec-0262bd41ec0f None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:04.372699 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f57f4f8-a69f-4643-baa2-73b49f178d98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:04.373295 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f57f4f8-a69f-4643-baa2-73b49f178d98 None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:04.373567 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f57f4f8-a69f-4643-baa2-73b49f178d98 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:04.373863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f57f4f8-a69f-4643-baa2-73b49f178d98 None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:04.374500 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f57f4f8-a69f-4643-baa2-73b49f178d98 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:04.375934 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f57f4f8-a69f-4643-baa2-73b49f178d98 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:04.400490 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 238/477] 10.222.0.26 () {60 vars in 1132 bytes} [Tue Sep 27 04:07:04 2022] POST /identity/v3/users => generated 249 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:07:04.413426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3cc3f1-bcff-4c64-9ca2-fc7ae65cdde9 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:04.514127 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:04.530789 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3cc3f1-bcff-4c64-9ca2-fc7ae65cdde9 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:04.567874 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c71a9d-2688-4a84-ad70-3ffd7d2c4323 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:04.568544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c71a9d-2688-4a84-ad70-3ffd7d2c4323 None admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:04.568885 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c71a9d-2688-4a84-ad70-3ffd7d2c4323 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:04.569156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c71a9d-2688-4a84-ad70-3ffd7d2c4323 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:04.569892 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c71a9d-2688-4a84-ad70-3ffd7d2c4323 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:04.571312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c71a9d-2688-4a84-ad70-3ffd7d2c4323 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:04.582886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.sql.core [None req-33c71a9d-2688-4a84-ad70-3ffd7d2c4323 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 27 04:07:04.582886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 27 04:07:04.582886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [parameters: {'id': 'e9cb32f569744dffa0a3e8399c6cec99', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 27 04:07:04.582886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81421) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 27 04:07:04.604914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-33c71a9d-2688-4a84-ad70-3ffd7d2c4323 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 27 04:07:04.604914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:04.604914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 27 04:07:04.604914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 27 04:07:04.604914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 27 04:07:04.604914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 27 04:07:04.604914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 27 04:07:04.604914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application result = self._query(query) Sep 27 04:07:04.604914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 27 04:07:04.604914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application conn.query(q) Sep 27 04:07:04.604914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 27 04:07:04.604914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 27 04:07:04.604914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 27 04:07:04.604914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application result.read() Sep 27 04:07:04.604914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 27 04:07:04.605870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 27 04:07:04.605870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 27 04:07:04.605870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 27 04:07:04.605870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 27 04:07:04.605870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 27 04:07:04.605870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 27 04:07:04.605870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 27 04:07:04.605870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 27 04:07:04.605870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:04.605870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 27 04:07:04.605870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:04.605870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:04.605870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 27 04:07:04.605870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 27 04:07:04.605870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Sep 27 04:07:04.605870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return ref.to_dict() Sep 27 04:07:04.605870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 27 04:07:04.605870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application next(self.gen) Sep 27 04:07:04.606864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 27 04:07:04.606864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application yield resource Sep 27 04:07:04.606864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 27 04:07:04.606864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application next(self.gen) Sep 27 04:07:04.606864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 27 04:07:04.606864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self.session.rollback() Sep 27 04:07:04.606864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 27 04:07:04.606864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self.force_reraise() Sep 27 04:07:04.606864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 27 04:07:04.606864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise self.value Sep 27 04:07:04.606864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 27 04:07:04.606864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 27 04:07:04.606864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 27 04:07:04.606864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application session.commit() Sep 27 04:07:04.606864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 27 04:07:04.606864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 27 04:07:04.606864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 27 04:07:04.607692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._prepare_impl() Sep 27 04:07:04.607692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 27 04:07:04.607692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self.session.flush() Sep 27 04:07:04.607692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 27 04:07:04.607692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._flush(objects) Sep 27 04:07:04.607692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 27 04:07:04.607692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 27 04:07:04.607692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 27 04:07:04.607692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application compat.raise_( Sep 27 04:07:04.607692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 27 04:07:04.607692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception Sep 27 04:07:04.607692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 27 04:07:04.607692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application flush_context.execute() Sep 27 04:07:04.607692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 27 04:07:04.607692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rec.execute(self) Sep 27 04:07:04.607692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 27 04:07:04.607692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 27 04:07:04.608566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 27 04:07:04.608566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 27 04:07:04.608566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 27 04:07:04.608566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 27 04:07:04.608566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 27 04:07:04.608566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 27 04:07:04.608566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 27 04:07:04.608566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 27 04:07:04.608566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 27 04:07:04.608566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 27 04:07:04.608566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 27 04:07:04.608566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 27 04:07:04.608566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 27 04:07:04.608566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 27 04:07:04.608566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 27 04:07:04.608566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception Sep 27 04:07:04.609379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 27 04:07:04.609379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 27 04:07:04.609379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 27 04:07:04.609379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 27 04:07:04.609379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 27 04:07:04.609379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application result = self._query(query) Sep 27 04:07:04.609379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 27 04:07:04.609379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application conn.query(q) Sep 27 04:07:04.609379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 27 04:07:04.609379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 27 04:07:04.609379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 27 04:07:04.609379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application result.read() Sep 27 04:07:04.609379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 27 04:07:04.609379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 27 04:07:04.609379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 27 04:07:04.609379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 27 04:07:04.611972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 27 04:07:04.611972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 27 04:07:04.611972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 27 04:07:04.611972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 27 04:07:04.611972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 27 04:07:04.611972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 27 04:07:04.611972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [parameters: {'id': 'e9cb32f569744dffa0a3e8399c6cec99', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 27 04:07:04.611972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 27 04:07:04.611972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:04.611972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:07:04.611972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:04.611972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:04.611972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:04.611972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:04.611972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:04.611972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:04.618438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:04.618438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:04.618438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:04.618438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:04.618438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:04.618438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:04.618438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Sep 27 04:07:04.618438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Sep 27 04:07:04.618438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:04.618438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:04.618438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Sep 27 04:07:04.618438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Sep 27 04:07:04.618438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 27 04:07:04.618438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 27 04:07:04.618438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 27 04:07:04.618438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:04.621408 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 240/478] 10.222.0.26 () {60 vars in 1131 bytes} [Tue Sep 27 04:07:04 2022] POST /identity/v3/roles => generated 144 bytes in 194 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:04.621560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f57f4f8-a69f-4643-baa2-73b49f178d98 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:04.681370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:04.743565 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f57f4f8-a69f-4643-baa2-73b49f178d98 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:04.788010 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4f6622-83d7-4856-ba3d-e28d9e1dbbce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:04.788558 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4f6622-83d7-4856-ba3d-e28d9e1dbbce None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:04.788832 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4f6622-83d7-4856-ba3d-e28d9e1dbbce None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:04.789100 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4f6622-83d7-4856-ba3d-e28d9e1dbbce None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:04.799729 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4f6622-83d7-4856-ba3d-e28d9e1dbbce None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:04.805491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-8e4f6622-83d7-4856-ba3d-e28d9e1dbbce None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:07:04.805491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:04.805491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:04.805491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:04.805491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:04.805491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:04.805491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:04.805491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:04.805491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:04.805491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:04.805491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:04.805491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:04.805491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:04.805491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:04.805491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:04.805491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:04.806401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:04.806401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:04.806401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:04.806401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:04.806401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:04.806401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:04.806401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:04.806401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:04.806401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:04.806401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:04.806401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:04.806401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:04.806401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:04.806401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:04.806401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:04.806401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:04.807178 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:04.807178 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:04.807178 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:04.807178 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:04.807178 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:04.807178 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:04.807178 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:04.807178 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:04.807178 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:04.807178 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:04.807178 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 04:07:04.807178 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:04.811596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 239/479] 10.222.0.26 () {56 vars in 1114 bytes} [Tue Sep 27 04:07:04 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:04.813022 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c71a9d-2688-4a84-ad70-3ffd7d2c4323 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:04.917327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c71a9d-2688-4a84-ad70-3ffd7d2c4323 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:04.952492 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f40dd9-d973-4119-a802-883792f91cce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:04.953088 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f40dd9-d973-4119-a802-883792f91cce None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:04.953362 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f40dd9-d973-4119-a802-883792f91cce None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:04.953630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f40dd9-d973-4119-a802-883792f91cce None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:04.954327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f40dd9-d973-4119-a802-883792f91cce None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:04.954628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f40dd9-d973-4119-a802-883792f91cce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:04.955970 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f40dd9-d973-4119-a802-883792f91cce None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:04.967723 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 241/480] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:04 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:05.685092 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cb4ba6-caf4-4e9a-96f5-1dc3dc1c2c71 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:05.685580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cb4ba6-caf4-4e9a-96f5-1dc3dc1c2c71 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:05.685580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cb4ba6-caf4-4e9a-96f5-1dc3dc1c2c71 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:05.687134 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 240/481] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:05.692681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdd56c6-907b-4f72-b284-c3a97103bde2 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:05.693319 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdd56c6-907b-4f72-b284-c3a97103bde2 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:05.693319 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdd56c6-907b-4f72-b284-c3a97103bde2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:05.722044 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-6bdd56c6-907b-4f72-b284-c3a97103bde2 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:05.723247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-6bdd56c6-907b-4f72-b284-c3a97103bde2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:05.735643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:05.820296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 242/482] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:07:06.201980 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cb4ba6-caf4-4e9a-96f5-1dc3dc1c2c71 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:06.335995 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edba45f-517d-4853-a150-23f62f55dcb2 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:06.336359 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edba45f-517d-4853-a150-23f62f55dcb2 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:06.336559 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edba45f-517d-4853-a150-23f62f55dcb2 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:06.338005 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 243/483] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:07:06.345787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdc754d-9c45-4bf0-a243-bf1c658e6047 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:06.345787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdc754d-9c45-4bf0-a243-bf1c658e6047 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:06.345787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdc754d-9c45-4bf0-a243-bf1c658e6047 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:06.350860 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cb4ba6-caf4-4e9a-96f5-1dc3dc1c2c71 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:06.361707 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:06.366826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-6fdc754d-9c45-4bf0-a243-bf1c658e6047 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:06.368702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-6fdc754d-9c45-4bf0-a243-bf1c658e6047 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:06.430455 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1c1a7b-a744-4d44-87a8-d97d99aa0ba7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:06.431111 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1c1a7b-a744-4d44-87a8-d97d99aa0ba7 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:06.431397 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1c1a7b-a744-4d44-87a8-d97d99aa0ba7 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:06.431671 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1c1a7b-a744-4d44-87a8-d97d99aa0ba7 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:06.454936 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 244/484] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:07:06.466372 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1c1a7b-a744-4d44-87a8-d97d99aa0ba7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:06.471277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-5f1c1a7b-a744-4d44-87a8-d97d99aa0ba7 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:06.471277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:06.471277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:06.471277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:06.471277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:06.471277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:06.471277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:06.471277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:06.471277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:06.471277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:06.471277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:06.471277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:06.471277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:06.471277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:06.471277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:06.471277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:06.472137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:06.472137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:06.472137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:06.472137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:06.472137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:06.472137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:06.472137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:06.472137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:06.472137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:06.472137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:06.472137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:06.472137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:06.472137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:06.472137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:06.472137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:06.472137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:06.472946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:06.472946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:06.472946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:06.472946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:06.472946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:06.472946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:06.472946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:06.472946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:06.472946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:06.472946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:06.472946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:06.472946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:06.472946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 241/485] 10.222.0.26 () {56 vars in 1102 bytes} [Tue Sep 27 04:07:06 2022] GET /identity/v3/roles/service => generated 85 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:06.478484 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fdc754d-9c45-4bf0-a243-bf1c658e6047 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:06.571358 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:06.613591 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fdc754d-9c45-4bf0-a243-bf1c658e6047 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:06.661980 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0135d95-8234-4afd-ae86-2e7ee001ef8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:06.662591 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0135d95-8234-4afd-ae86-2e7ee001ef8d None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:06.662873 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0135d95-8234-4afd-ae86-2e7ee001ef8d None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:06.663149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0135d95-8234-4afd-ae86-2e7ee001ef8d None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:06.663824 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0135d95-8234-4afd-ae86-2e7ee001ef8d None admin] RBAC: Adding query filter params (name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:06.664128 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0135d95-8234-4afd-ae86-2e7ee001ef8d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:06.665485 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0135d95-8234-4afd-ae86-2e7ee001ef8d None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:06.677947 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 245/486] 10.222.0.26 () {56 vars in 1103 bytes} [Tue Sep 27 04:07:06 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:06.683481 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1c1a7b-a744-4d44-87a8-d97d99aa0ba7 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:06.840474 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0135d95-8234-4afd-ae86-2e7ee001ef8d None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:06.846050 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1c1a7b-a744-4d44-87a8-d97d99aa0ba7 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:06.888779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0508e02-946a-41ab-be6b-c96e0bbc301c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:06.889919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0508e02-946a-41ab-be6b-c96e0bbc301c None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:06.890195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0508e02-946a-41ab-be6b-c96e0bbc301c None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:06.890465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0508e02-946a-41ab-be6b-c96e0bbc301c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:06.905565 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0508e02-946a-41ab-be6b-c96e0bbc301c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:06.907740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0508e02-946a-41ab-be6b-c96e0bbc301c None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:06.926297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-f0508e02-946a-41ab-be6b-c96e0bbc301c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:06.926297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:06.926297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:06.926297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:06.926297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:06.926297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:06.926297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:06.926297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:06.926297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:06.926297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:06.926297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:06.926297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:06.926297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:06.926297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:06.926297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:06.926297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:06.927861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:06.927861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:06.927861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:06.927861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:06.927861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:06.927861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:06.927861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:06.927861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:06.927861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:06.927861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:06.927861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:06.927861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:06.927861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:06.927861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:06.927861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:06.927861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:06.927861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:06.928688 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:06.928688 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:06.928688 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:06.928857 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 242/487] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:06.930271 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82a734c-dd7d-4cdc-bbe8-64f79a36b38f None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:06.930955 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82a734c-dd7d-4cdc-bbe8-64f79a36b38f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:06.931237 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82a734c-dd7d-4cdc-bbe8-64f79a36b38f None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:06.933331 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 243/488] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:06.934371 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82a734c-dd7d-4cdc-bbe8-64f79a36b38f None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:06.957773 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0135d95-8234-4afd-ae86-2e7ee001ef8d None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:06.960942 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:06.999205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19471b7f-d392-4491-b20c-039033a7ef45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:07.000420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19471b7f-d392-4491-b20c-039033a7ef45 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:07.001077 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19471b7f-d392-4491-b20c-039033a7ef45 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:07.001446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19471b7f-d392-4491-b20c-039033a7ef45 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:07.003905 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:07.015473 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19471b7f-d392-4491-b20c-039033a7ef45 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:07.021961 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-19471b7f-d392-4491-b20c-039033a7ef45 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:07.021961 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:07.021961 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:07.021961 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:07.021961 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:07.021961 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:07.021961 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:07.021961 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:07.021961 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:07.021961 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:07.021961 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:07.021961 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:07.021961 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:07.021961 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:07.021961 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:07.021961 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:07.023688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:07.023688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:07.023688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:07.023688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:07.023688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:07.023688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:07.023688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:07.023688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:07.023688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:07.023688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:07.023688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:07.023688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:07.023688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:07.023688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:07.023688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:07.023688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:07.025167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:07.025167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:07.025167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:07.025167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:07.025167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:07.025167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:07.025167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:07.025167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:07.025167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:07.025167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:07.025167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:07.025167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:07.025167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 246/489] 10.222.0.26 () {56 vars in 1102 bytes} [Tue Sep 27 04:07:06 2022] GET /identity/v3/roles/service => generated 85 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:07:07.031299 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996ffb5b-e252-468d-acfa-23cfa8dc73a0 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:07.031483 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996ffb5b-e252-468d-acfa-23cfa8dc73a0 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:07.031774 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996ffb5b-e252-468d-acfa-23cfa8dc73a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:07.055255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-996ffb5b-e252-468d-acfa-23cfa8dc73a0 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:07.056751 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-996ffb5b-e252-468d-acfa-23cfa8dc73a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:07.068313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82a734c-dd7d-4cdc-bbe8-64f79a36b38f None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:07.112869 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf480ee-16fd-463a-a700-a92459e68a86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:07.113646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf480ee-16fd-463a-a700-a92459e68a86 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:07.113951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf480ee-16fd-463a-a700-a92459e68a86 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:07.114253 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf480ee-16fd-463a-a700-a92459e68a86 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:07.115141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf480ee-16fd-463a-a700-a92459e68a86 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:07.115439 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf480ee-16fd-463a-a700-a92459e68a86 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:07.117184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf480ee-16fd-463a-a700-a92459e68a86 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:07.133061 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 244/490] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:07.135429 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 247/491] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:07:07.138407 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf480ee-16fd-463a-a700-a92459e68a86 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:07.140422 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996ffb5b-e252-468d-acfa-23cfa8dc73a0 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:07.276244 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:07.307239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf480ee-16fd-463a-a700-a92459e68a86 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:07.325210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996ffb5b-e252-468d-acfa-23cfa8dc73a0 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:07.368123 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:07.368977 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b7ce8e-2507-4643-9509-bc6c78877dc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:07.369577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b7ce8e-2507-4643-9509-bc6c78877dc2 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:07.370023 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b7ce8e-2507-4643-9509-bc6c78877dc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:07.370296 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b7ce8e-2507-4643-9509-bc6c78877dc2 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:07.371066 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b7ce8e-2507-4643-9509-bc6c78877dc2 None admin] RBAC: Adding query filter params (name=service) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:07.371360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b7ce8e-2507-4643-9509-bc6c78877dc2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:07.373445 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b7ce8e-2507-4643-9509-bc6c78877dc2 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:07.389872 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d18d95-78ae-4e55-940f-5d142f856035 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:07.390493 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d18d95-78ae-4e55-940f-5d142f856035 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:07.390767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d18d95-78ae-4e55-940f-5d142f856035 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:07.391032 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d18d95-78ae-4e55-940f-5d142f856035 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:07.392136 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 245/492] 10.222.0.26 () {56 vars in 1103 bytes} [Tue Sep 27 04:07:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:07.398760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b7ce8e-2507-4643-9509-bc6c78877dc2 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:07.404758 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d18d95-78ae-4e55-940f-5d142f856035 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:07.405978 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d18d95-78ae-4e55-940f-5d142f856035 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:07.432025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-e3d18d95-78ae-4e55-940f-5d142f856035 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 27 04:07:07.432025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:07.432025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:07.432025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:07.432025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:07.432025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:07.432025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:07.432025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:07.432025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:07.432025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:07.432025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:07.432025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:07.432025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:07.432025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:07.432025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:07.432025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:07.432957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:07.432957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:07.432957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:07.432957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:07.432957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:07.432957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:07.432957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:07.432957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:07.432957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:07.432957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:07.432957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:07.432957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:07.432957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:07.432957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:07.432957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:07.432957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:07.432957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:07.432957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:07.433868 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:07.433868 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:07.433868 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:07.433868 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:07.433868 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:07.433868 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:07.433868 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:07.433868 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:07.433868 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 27 04:07:07.433868 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:07.439316 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 248/493] 10.222.0.26 () {56 vars in 1093 bytes} [Tue Sep 27 04:07:07 2022] GET /identity/v3/users/nova => generated 82 bytes in 293 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:07.441205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d18d95-78ae-4e55-940f-5d142f856035 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:07.538687 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b7ce8e-2507-4643-9509-bc6c78877dc2 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:07.554698 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d18d95-78ae-4e55-940f-5d142f856035 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:07.578175 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de6c162-7017-459b-a2ff-18390891f266 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:07.578916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de6c162-7017-459b-a2ff-18390891f266 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:07.579264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de6c162-7017-459b-a2ff-18390891f266 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:07.579623 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de6c162-7017-459b-a2ff-18390891f266 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:07.590602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:07.593014 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de6c162-7017-459b-a2ff-18390891f266 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:07.594663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de6c162-7017-459b-a2ff-18390891f266 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:07.596245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f1dca7-59db-43ec-80ef-55e71388e506 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:07.597508 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f1dca7-59db-43ec-80ef-55e71388e506 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:07.598045 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f1dca7-59db-43ec-80ef-55e71388e506 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:07.598461 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f1dca7-59db-43ec-80ef-55e71388e506 None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:07.599471 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f1dca7-59db-43ec-80ef-55e71388e506 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:07.599781 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f1dca7-59db-43ec-80ef-55e71388e506 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:07.601196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f1dca7-59db-43ec-80ef-55e71388e506 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:07.610355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-8de6c162-7017-459b-a2ff-18390891f266 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:07.610355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:07.610355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:07.610355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:07.610355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:07.610355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:07.610355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:07.610355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:07.610355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:07.610355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:07.610355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:07.610355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:07.610355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:07.610355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:07.610355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:07.610355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:07.611309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:07.611309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:07.611309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:07.611309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:07.611309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:07.611309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:07.611309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:07.611309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:07.611309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:07.611309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:07.611309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:07.611309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:07.611309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:07.611309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:07.611309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:07.611309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:07.611309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:07.612130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:07.612130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:07.612130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:07.615979 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 246/494] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:07.616748 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de6c162-7017-459b-a2ff-18390891f266 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:07.644530 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 249/495] 10.222.0.26 () {56 vars in 1133 bytes} [Tue Sep 27 04:07:07 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:07.644530 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a3d2f3-af1e-4c0f-b3d9-d33f54b85f5c None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:07.644530 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a3d2f3-af1e-4c0f-b3d9-d33f54b85f5c None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:07.644530 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a3d2f3-af1e-4c0f-b3d9-d33f54b85f5c None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:07.644530 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 250/496] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:07.644530 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a3d2f3-af1e-4c0f-b3d9-d33f54b85f5c None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:07.691193 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:07.735541 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de6c162-7017-459b-a2ff-18390891f266 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:07.747922 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a3d2f3-af1e-4c0f-b3d9-d33f54b85f5c None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:07.769143 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40dd3a37-a765-4759-9013-a90c76f7e345 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:07.769774 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40dd3a37-a765-4759-9013-a90c76f7e345 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:07.770052 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40dd3a37-a765-4759-9013-a90c76f7e345 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:07.770321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40dd3a37-a765-4759-9013-a90c76f7e345 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:07.779864 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40dd3a37-a765-4759-9013-a90c76f7e345 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:07.780404 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b656dd81-89d3-4ae0-89d3-9802308bdcbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:07.780994 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b656dd81-89d3-4ae0-89d3-9802308bdcbe None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:07.781272 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b656dd81-89d3-4ae0-89d3-9802308bdcbe None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:07.781544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b656dd81-89d3-4ae0-89d3-9802308bdcbe None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:07.782213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b656dd81-89d3-4ae0-89d3-9802308bdcbe None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:07.782530 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b656dd81-89d3-4ae0-89d3-9802308bdcbe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:07.783871 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b656dd81-89d3-4ae0-89d3-9802308bdcbe None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:07.784998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-40dd3a37-a765-4759-9013-a90c76f7e345 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:07.784998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:07.784998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:07.784998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:07.784998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:07.784998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:07.784998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:07.784998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:07.784998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:07.784998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:07.784998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:07.784998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:07.784998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:07.784998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:07.784998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:07.784998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:07.785842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:07.785842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:07.785842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:07.785842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:07.785842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:07.785842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:07.785842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:07.785842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:07.785842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:07.785842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:07.785842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:07.785842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:07.785842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:07.785842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:07.785842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:07.785842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:07.787141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:07.787141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:07.787141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:07.787141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:07.787141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:07.787141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:07.787141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:07.787141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:07.787141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:07.787141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:07.787141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:07.787141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:07.787141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 247/497] 10.222.0.26 () {56 vars in 1102 bytes} [Tue Sep 27 04:07:07 2022] GET /identity/v3/roles/service => generated 85 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:07.787796 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40dd3a37-a765-4759-9013-a90c76f7e345 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:07.801249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 251/498] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:07.801403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ebc5f3-f549-4db3-9259-6e63fabbded5 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:07.801688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ebc5f3-f549-4db3-9259-6e63fabbded5 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:07.802546 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ebc5f3-f549-4db3-9259-6e63fabbded5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:07.821088 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-62ebc5f3-f549-4db3-9259-6e63fabbded5 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:07.822403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-62ebc5f3-f549-4db3-9259-6e63fabbded5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:07.840350 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:07.887432 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40dd3a37-a765-4759-9013-a90c76f7e345 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:07.888434 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 252/499] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:07:07.889650 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ebeedd-e053-4cc9-9533-42152fae9ff7 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:07.889954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ebeedd-e053-4cc9-9533-42152fae9ff7 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:07.890222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ebeedd-e053-4cc9-9533-42152fae9ff7 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:07.891416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 253/500] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:07 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:07.892093 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ebeedd-e053-4cc9-9533-42152fae9ff7 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:07.931967 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:07.934416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad800b26-9452-4200-b819-2a04e054a2bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:07.935063 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad800b26-9452-4200-b819-2a04e054a2bb None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:07.935284 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad800b26-9452-4200-b819-2a04e054a2bb None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:07.935547 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad800b26-9452-4200-b819-2a04e054a2bb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:08.006337 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad800b26-9452-4200-b819-2a04e054a2bb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:08.007652 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ebeedd-e053-4cc9-9533-42152fae9ff7 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:08.007849 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad800b26-9452-4200-b819-2a04e054a2bb None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:08.049714 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822887a2-eaff-469a-8001-a68c4993bfe9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:08.050861 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822887a2-eaff-469a-8001-a68c4993bfe9 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:08.051810 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822887a2-eaff-469a-8001-a68c4993bfe9 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:08.052417 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822887a2-eaff-469a-8001-a68c4993bfe9 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:08.053407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822887a2-eaff-469a-8001-a68c4993bfe9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:08.054060 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822887a2-eaff-469a-8001-a68c4993bfe9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:08.055872 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822887a2-eaff-469a-8001-a68c4993bfe9 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:08.057502 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:08.067292 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 254/501] 10.222.0.26 () {56 vars in 1103 bytes} [Tue Sep 27 04:07:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:08.068125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822887a2-eaff-469a-8001-a68c4993bfe9 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:08.115068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 248/502] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:07:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:07:08.115068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6467eb7e-c85b-4e6a-91bd-6ebf88ff5ea7 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:08.115068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6467eb7e-c85b-4e6a-91bd-6ebf88ff5ea7 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:08.115068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6467eb7e-c85b-4e6a-91bd-6ebf88ff5ea7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:08.131454 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-6467eb7e-c85b-4e6a-91bd-6ebf88ff5ea7 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:08.132637 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6467eb7e-c85b-4e6a-91bd-6ebf88ff5ea7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:08.173331 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:08.180567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822887a2-eaff-469a-8001-a68c4993bfe9 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:08.199530 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 249/503] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:07:08.200300 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6467eb7e-c85b-4e6a-91bd-6ebf88ff5ea7 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:08.217699 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4352339c-2702-4d37-bbfc-007d9bbd5517 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:08.218356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4352339c-2702-4d37-bbfc-007d9bbd5517 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:08.218656 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4352339c-2702-4d37-bbfc-007d9bbd5517 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:08.218900 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4352339c-2702-4d37-bbfc-007d9bbd5517 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:08.226745 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4352339c-2702-4d37-bbfc-007d9bbd5517 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:08.228152 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4352339c-2702-4d37-bbfc-007d9bbd5517 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:08.239409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-4352339c-2702-4d37-bbfc-007d9bbd5517 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 27 04:07:08.239409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:08.239409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:08.239409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:08.239409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:08.239409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:08.239409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:08.239409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:08.239409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:08.239409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:08.239409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:08.239409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:08.239409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:08.239409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:08.239409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:08.239409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:08.240300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:08.240300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:08.240300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:08.240300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:08.240300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:08.240300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:08.240300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:08.240300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:08.240300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:08.240300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:08.240300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:08.240300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:08.240300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:08.240300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:08.240300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:08.240300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:08.240300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:08.240300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:08.241191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:08.241191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:08.241191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:08.241191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:08.241191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:08.241191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:08.241191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:08.241191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:08.241191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 27 04:07:08.241191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:08.241191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 255/504] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:07:08 2022] GET /identity/v3/users/glance => generated 84 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:08.241191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4352339c-2702-4d37-bbfc-007d9bbd5517 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:08.288798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:08.308584 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6467eb7e-c85b-4e6a-91bd-6ebf88ff5ea7 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:08.343044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c19b7b-ced8-46e8-babf-57eb26b6419a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:08.343683 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c19b7b-ced8-46e8-babf-57eb26b6419a None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:08.343956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c19b7b-ced8-46e8-babf-57eb26b6419a None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:08.344219 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c19b7b-ced8-46e8-babf-57eb26b6419a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:08.348149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4352339c-2702-4d37-bbfc-007d9bbd5517 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:08.354051 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c19b7b-ced8-46e8-babf-57eb26b6419a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:08.355521 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c19b7b-ced8-46e8-babf-57eb26b6419a None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:08.370145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-80c19b7b-ced8-46e8-babf-57eb26b6419a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:08.370145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:08.370145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:08.370145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:08.370145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:08.370145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:08.370145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:08.370145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:08.370145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:08.370145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:08.370145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:08.370145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:08.370145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:08.370145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:08.370145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:08.370145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:08.371043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:08.371043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:08.371043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:08.371043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:08.371043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:08.371043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:08.371043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:08.371043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:08.371043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:08.371043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:08.371043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:08.371043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:08.371043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:08.371043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:08.371043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:08.371043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:08.371043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:08.371859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:08.371859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:08.371859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:08.371859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 250/505] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:08.372151 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c19b7b-ced8-46e8-babf-57eb26b6419a None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:08.388121 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d3db3b-29f9-4cc9-bc2e-752a5ab53f4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:08.388749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d3db3b-29f9-4cc9-bc2e-752a5ab53f4a None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:08.389032 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d3db3b-29f9-4cc9-bc2e-752a5ab53f4a None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:08.389308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d3db3b-29f9-4cc9-bc2e-752a5ab53f4a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:08.401462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d3db3b-29f9-4cc9-bc2e-752a5ab53f4a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:08.404928 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d3db3b-29f9-4cc9-bc2e-752a5ab53f4a None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:08.418426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-14d3db3b-29f9-4cc9-bc2e-752a5ab53f4a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:08.418426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:08.418426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:08.418426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:08.418426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:08.418426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:08.418426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:08.418426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:08.418426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:08.418426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:08.418426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:08.418426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:08.418426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:08.418426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:08.418426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:08.418426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:08.419353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:08.419353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:08.419353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:08.419353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:08.419353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:08.419353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:08.419353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:08.419353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:08.419353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:08.419353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:08.419353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:08.419353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:08.419353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:08.419353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:08.419353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:08.419353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:08.419353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:08.420184 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:08.420184 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:08.420184 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:08.420184 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 256/506] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:08.420667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d3db3b-29f9-4cc9-bc2e-752a5ab53f4a None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:08.433507 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:08.482213 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c19b7b-ced8-46e8-babf-57eb26b6419a None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:08.514790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980035a0-d2fa-467e-a447-16a5b097ee09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:08.515405 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980035a0-d2fa-467e-a447-16a5b097ee09 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:08.515681 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980035a0-d2fa-467e-a447-16a5b097ee09 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:08.515953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980035a0-d2fa-467e-a447-16a5b097ee09 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:08.517735 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:08.527073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d3db3b-29f9-4cc9-bc2e-752a5ab53f4a None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:08.527351 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980035a0-d2fa-467e-a447-16a5b097ee09 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:08.533206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-980035a0-d2fa-467e-a447-16a5b097ee09 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:08.533206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:08.533206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:08.533206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:08.533206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:08.533206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:08.533206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:08.533206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:08.533206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:08.533206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:08.533206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:08.533206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:08.533206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:08.533206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:08.533206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:08.533206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:08.534135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:08.534135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:08.534135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:08.534135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:08.534135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:08.534135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:08.534135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:08.534135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:08.534135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:08.534135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:08.534135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:08.534135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:08.534135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:08.534135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:08.534135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:08.534135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:08.534912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:08.534912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:08.534912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:08.534912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:08.534912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:08.534912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:08.534912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:08.534912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:08.534912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:08.534912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:08.534912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:08.534912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:08.534912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 251/507] 10.222.0.26 () {56 vars in 1102 bytes} [Tue Sep 27 04:07:08 2022] GET /identity/v3/roles/service => generated 85 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:08.535569 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980035a0-d2fa-467e-a447-16a5b097ee09 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:08.562888 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd721b0c-210f-417f-9f1f-83c7a8b4e159 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:08.563480 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd721b0c-210f-417f-9f1f-83c7a8b4e159 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:08.563759 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd721b0c-210f-417f-9f1f-83c7a8b4e159 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:08.564031 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd721b0c-210f-417f-9f1f-83c7a8b4e159 None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:08.564747 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd721b0c-210f-417f-9f1f-83c7a8b4e159 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:08.565052 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd721b0c-210f-417f-9f1f-83c7a8b4e159 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:08.566506 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd721b0c-210f-417f-9f1f-83c7a8b4e159 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:08.583673 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 257/508] 10.222.0.26 () {56 vars in 1137 bytes} [Tue Sep 27 04:07:08 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:08.584453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd721b0c-210f-417f-9f1f-83c7a8b4e159 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:08.634816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980035a0-d2fa-467e-a447-16a5b097ee09 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:08.646446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:08.668044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dcec66-b38a-48e2-a708-5ee1479d6e60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:08.668634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dcec66-b38a-48e2-a708-5ee1479d6e60 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:08.668910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dcec66-b38a-48e2-a708-5ee1479d6e60 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:08.669176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dcec66-b38a-48e2-a708-5ee1479d6e60 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:08.669841 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dcec66-b38a-48e2-a708-5ee1479d6e60 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:08.670138 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dcec66-b38a-48e2-a708-5ee1479d6e60 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:08.671511 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dcec66-b38a-48e2-a708-5ee1479d6e60 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:08.683188 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 252/509] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:08.683693 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd721b0c-210f-417f-9f1f-83c7a8b4e159 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:08.683988 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dcec66-b38a-48e2-a708-5ee1479d6e60 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:08.720046 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b796d4-8896-4495-8348-8b193bf9a4d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:08.720648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b796d4-8896-4495-8348-8b193bf9a4d8 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:08.720927 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b796d4-8896-4495-8348-8b193bf9a4d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:08.721200 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b796d4-8896-4495-8348-8b193bf9a4d8 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:08.721878 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b796d4-8896-4495-8348-8b193bf9a4d8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:08.722177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b796d4-8896-4495-8348-8b193bf9a4d8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:08.723529 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b796d4-8896-4495-8348-8b193bf9a4d8 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:08.734557 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 258/510] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:08.735273 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b796d4-8896-4495-8348-8b193bf9a4d8 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:08.743074 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:08.787268 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dcec66-b38a-48e2-a708-5ee1479d6e60 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:08.820136 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b05656-1ec1-4ab3-857d-f9de0b6e4b4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:08.820753 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b05656-1ec1-4ab3-857d-f9de0b6e4b4f None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:08.821023 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b05656-1ec1-4ab3-857d-f9de0b6e4b4f None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:08.821285 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b05656-1ec1-4ab3-857d-f9de0b6e4b4f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:08.830882 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b05656-1ec1-4ab3-857d-f9de0b6e4b4f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:08.832091 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b796d4-8896-4495-8348-8b193bf9a4d8 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:08.832337 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b05656-1ec1-4ab3-857d-f9de0b6e4b4f None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:08.846742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-b0b05656-1ec1-4ab3-857d-f9de0b6e4b4f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:08.846742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:08.846742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:08.846742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:08.846742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:08.846742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:08.846742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:08.846742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:08.846742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:08.846742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:08.846742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:08.846742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:08.846742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:08.846742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:08.846742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:08.846742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:08.847650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:08.847650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:08.847650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:08.847650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:08.847650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:08.847650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:08.847650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:08.847650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:08.847650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:08.847650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:08.847650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:08.847650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:08.847650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:08.847650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:08.847650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:08.847650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:08.847650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:08.848499 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:08.848499 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:08.848499 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:08.848499 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 253/511] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:08.848749 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b05656-1ec1-4ab3-857d-f9de0b6e4b4f None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:08.856362 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:08.867614 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e62d2c3-92e4-46f5-89e9-e6a872615761 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:08.868213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e62d2c3-92e4-46f5-89e9-e6a872615761 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:08.868495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e62d2c3-92e4-46f5-89e9-e6a872615761 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:08.868769 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e62d2c3-92e4-46f5-89e9-e6a872615761 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:08.869426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e62d2c3-92e4-46f5-89e9-e6a872615761 None admin] RBAC: Adding query filter params (name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:08.869729 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e62d2c3-92e4-46f5-89e9-e6a872615761 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:08.871108 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e62d2c3-92e4-46f5-89e9-e6a872615761 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:08.879222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 259/512] 10.222.0.26 () {56 vars in 1103 bytes} [Tue Sep 27 04:07:08 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:08.880548 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e62d2c3-92e4-46f5-89e9-e6a872615761 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:08.946366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b05656-1ec1-4ab3-857d-f9de0b6e4b4f None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:08.953127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:08.979496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e021eaf-3200-4577-85e4-ed5e517935e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:08.980092 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e021eaf-3200-4577-85e4-ed5e517935e3 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:08.980369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e021eaf-3200-4577-85e4-ed5e517935e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:08.980640 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e021eaf-3200-4577-85e4-ed5e517935e3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:08.984522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e62d2c3-92e4-46f5-89e9-e6a872615761 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:09.020608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2526b9-5556-4bd9-bd8c-c46eb27cfec4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:09.021155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2526b9-5556-4bd9-bd8c-c46eb27cfec4 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:09.021440 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2526b9-5556-4bd9-bd8c-c46eb27cfec4 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:09.021730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2526b9-5556-4bd9-bd8c-c46eb27cfec4 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:09.029363 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2526b9-5556-4bd9-bd8c-c46eb27cfec4 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:09.030831 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2526b9-5556-4bd9-bd8c-c46eb27cfec4 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:09.042202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-0e2526b9-5556-4bd9-bd8c-c46eb27cfec4 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 27 04:07:09.042202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:09.042202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:09.042202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:09.042202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:09.042202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:09.042202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:09.042202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:09.042202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:09.042202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:09.042202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:09.042202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:09.042202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:09.042202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:09.042202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:09.042202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:09.043170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:09.043170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:09.043170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:09.043170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:09.043170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:09.043170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:09.043170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:09.043170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:09.043170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:09.043170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:09.043170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:09.043170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:09.043170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:09.043170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:09.043170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:09.043170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:09.043170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:09.043170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:09.044078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:09.044078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:09.044078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:09.044078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:09.044078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:09.044078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:09.044078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:09.044078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:09.044078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 27 04:07:09.044078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:09.044078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 260/513] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:07:08 2022] GET /identity/v3/users/cinder => generated 84 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:09.044078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2526b9-5556-4bd9-bd8c-c46eb27cfec4 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:09.052655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e021eaf-3200-4577-85e4-ed5e517935e3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:09.054150 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e021eaf-3200-4577-85e4-ed5e517935e3 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:09.092471 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:09.145316 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2526b9-5556-4bd9-bd8c-c46eb27cfec4 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:09.148288 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 254/514] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:07:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:07:09.149101 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e021eaf-3200-4577-85e4-ed5e517935e3 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:09.156981 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:09.180441 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49ec34a-baff-4e4d-a910-35b49002d48e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:09.181062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49ec34a-baff-4e4d-a910-35b49002d48e None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:09.181340 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49ec34a-baff-4e4d-a910-35b49002d48e None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:09.181610 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49ec34a-baff-4e4d-a910-35b49002d48e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:09.188081 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49ec34a-baff-4e4d-a910-35b49002d48e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:09.189556 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49ec34a-baff-4e4d-a910-35b49002d48e None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:09.201299 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-e49ec34a-baff-4e4d-a910-35b49002d48e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:09.201299 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:09.201299 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:09.201299 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:09.201299 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:09.201299 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:09.201299 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:09.201299 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:09.201299 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:09.201299 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:09.201299 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:09.201299 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:09.201299 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:09.201299 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:09.201299 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:09.202324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:09.202324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:09.202324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:09.202324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:09.202324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:09.202324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:09.202324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:09.202324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:09.202324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:09.202324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:09.202324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:09.202324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:09.202324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:09.202324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:09.202324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:09.202324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:09.202324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:09.203158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:09.203158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:09.203158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:09.203158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:09.203158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:09.203158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:09.203158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:09.203158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:09.203158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 261/515] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:09.203158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49ec34a-baff-4e4d-a910-35b49002d48e None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:09.249459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e021eaf-3200-4577-85e4-ed5e517935e3 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:09.281767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632de5ab-d798-4052-b849-4de9a3058666 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:09.282670 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632de5ab-d798-4052-b849-4de9a3058666 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:09.283103 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632de5ab-d798-4052-b849-4de9a3058666 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:09.283376 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632de5ab-d798-4052-b849-4de9a3058666 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:09.284013 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632de5ab-d798-4052-b849-4de9a3058666 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:09.284312 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632de5ab-d798-4052-b849-4de9a3058666 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:09.285673 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632de5ab-d798-4052-b849-4de9a3058666 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:09.296852 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 255/516] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:09.297627 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632de5ab-d798-4052-b849-4de9a3058666 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:09.299229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49ec34a-baff-4e4d-a910-35b49002d48e None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:09.306570 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:09.332603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea300ba-d76e-4892-b4a3-2f360f39bb01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:09.333795 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea300ba-d76e-4892-b4a3-2f360f39bb01 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:09.334073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea300ba-d76e-4892-b4a3-2f360f39bb01 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:09.334341 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea300ba-d76e-4892-b4a3-2f360f39bb01 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:09.345073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea300ba-d76e-4892-b4a3-2f360f39bb01 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:09.346505 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea300ba-d76e-4892-b4a3-2f360f39bb01 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:09.360456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:09.360648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-bea300ba-d76e-4892-b4a3-2f360f39bb01 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:09.360648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:09.360648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:09.360648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:09.360648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:09.360648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:09.360648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:09.360648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:09.360648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:09.360648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:09.360648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:09.360648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:09.360648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:09.360648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:09.360648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:09.360648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:09.361553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:09.361553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:09.361553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:09.361553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:09.361553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:09.361553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:09.361553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:09.361553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:09.361553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:09.361553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:09.361553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:09.361553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:09.361553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:09.361553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:09.361553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:09.361553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:09.361553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:09.362491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:09.362491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:09.362491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:09.362491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 262/517] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:09.362705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea300ba-d76e-4892-b4a3-2f360f39bb01 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:09.397041 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632de5ab-d798-4052-b849-4de9a3058666 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:09.431841 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72528477-f00d-4a17-9cc1-8d4eeb5e74c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:09.432437 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72528477-f00d-4a17-9cc1-8d4eeb5e74c5 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:09.432709 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72528477-f00d-4a17-9cc1-8d4eeb5e74c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:09.432981 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72528477-f00d-4a17-9cc1-8d4eeb5e74c5 None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:09.433699 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72528477-f00d-4a17-9cc1-8d4eeb5e74c5 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:09.434034 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72528477-f00d-4a17-9cc1-8d4eeb5e74c5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:09.435446 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72528477-f00d-4a17-9cc1-8d4eeb5e74c5 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:09.463935 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea300ba-d76e-4892-b4a3-2f360f39bb01 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:09.470597 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 256/518] 10.222.0.26 () {56 vars in 1137 bytes} [Tue Sep 27 04:07:09 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:09.471402 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72528477-f00d-4a17-9cc1-8d4eeb5e74c5 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:09.505132 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8523a8ee-ab3c-4a13-8e45-d91b807ac682 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:09.505783 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8523a8ee-ab3c-4a13-8e45-d91b807ac682 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:09.506062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8523a8ee-ab3c-4a13-8e45-d91b807ac682 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:09.506337 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8523a8ee-ab3c-4a13-8e45-d91b807ac682 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:09.516599 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8523a8ee-ab3c-4a13-8e45-d91b807ac682 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:09.518059 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8523a8ee-ab3c-4a13-8e45-d91b807ac682 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:09.532310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-8523a8ee-ab3c-4a13-8e45-d91b807ac682 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:09.532310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:09.532310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:09.532310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:09.532310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:09.532310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:09.532310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:09.532310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:09.532310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:09.532310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:09.532310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:09.532310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:09.532310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:09.532310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:09.532310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:09.532310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:09.533385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:09.533385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:09.533385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:09.533385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:09.533385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:09.533385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:09.533385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:09.533385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:09.533385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:09.533385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:09.533385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:09.533385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:09.533385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:09.533385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:09.533385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:09.533385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:09.533385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:09.534439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:09.534439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:09.534439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:09.534439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 263/519] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:09.534439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8523a8ee-ab3c-4a13-8e45-d91b807ac682 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:09.547421 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:09.571597 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72528477-f00d-4a17-9cc1-8d4eeb5e74c5 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:09.594257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:09.605106 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f65f9a8-df72-45fb-a429-2478a44a9a59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:09.605692 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f65f9a8-df72-45fb-a429-2478a44a9a59 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:09.605998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f65f9a8-df72-45fb-a429-2478a44a9a59 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:09.606266 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f65f9a8-df72-45fb-a429-2478a44a9a59 None admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:09.606945 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f65f9a8-df72-45fb-a429-2478a44a9a59 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:09.607244 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f65f9a8-df72-45fb-a429-2478a44a9a59 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:09.608666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f65f9a8-df72-45fb-a429-2478a44a9a59 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:09.625237 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 257/520] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:09.626037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f65f9a8-df72-45fb-a429-2478a44a9a59 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:09.633315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8523a8ee-ab3c-4a13-8e45-d91b807ac682 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:09.666347 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e817424-1c04-40c3-a9d7-4a505dd494bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:09.666941 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e817424-1c04-40c3-a9d7-4a505dd494bf None admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:09.667228 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e817424-1c04-40c3-a9d7-4a505dd494bf None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:09.667498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e817424-1c04-40c3-a9d7-4a505dd494bf None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:09.668131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e817424-1c04-40c3-a9d7-4a505dd494bf None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:09.669554 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e817424-1c04-40c3-a9d7-4a505dd494bf None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:09.690785 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 264/521] 10.222.0.26 () {60 vars in 1132 bytes} [Tue Sep 27 04:07:09 2022] POST /identity/v3/users => generated 247 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:07:09.691743 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e817424-1c04-40c3-a9d7-4a505dd494bf None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:09.730500 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f65f9a8-df72-45fb-a429-2478a44a9a59 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:09.768758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21171b63-f752-427f-845b-17994774e150 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:09.769372 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21171b63-f752-427f-845b-17994774e150 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:09.769648 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21171b63-f752-427f-845b-17994774e150 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:09.769942 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21171b63-f752-427f-845b-17994774e150 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:09.771247 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21171b63-f752-427f-845b-17994774e150 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:09.771545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21171b63-f752-427f-845b-17994774e150 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:09.772980 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21171b63-f752-427f-845b-17994774e150 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:09.779811 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:09.790574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 258/522] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:09.790574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21171b63-f752-427f-845b-17994774e150 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:09.827638 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e817424-1c04-40c3-a9d7-4a505dd494bf None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:09.838507 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:09.867631 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a41da0-a908-4b13-a623-89598c2bf508 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:09.868244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a41da0-a908-4b13-a623-89598c2bf508 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:09.868529 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a41da0-a908-4b13-a623-89598c2bf508 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:09.868810 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a41da0-a908-4b13-a623-89598c2bf508 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:09.902238 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21171b63-f752-427f-845b-17994774e150 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:09.941189 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1665d64d-c786-46c2-9ea8-1103d37d3b88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:09.941834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1665d64d-c786-46c2-9ea8-1103d37d3b88 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:09.942138 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1665d64d-c786-46c2-9ea8-1103d37d3b88 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:09.942925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1665d64d-c786-46c2-9ea8-1103d37d3b88 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:09.943578 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1665d64d-c786-46c2-9ea8-1103d37d3b88 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:09.943886 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1665d64d-c786-46c2-9ea8-1103d37d3b88 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:09.945308 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1665d64d-c786-46c2-9ea8-1103d37d3b88 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:09.950439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a41da0-a908-4b13-a623-89598c2bf508 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:09.951904 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a41da0-a908-4b13-a623-89598c2bf508 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:09.957686 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 259/523] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:09.958948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1665d64d-c786-46c2-9ea8-1103d37d3b88 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:10.022467 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:10.068500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 265/524] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:07:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:07:10.068665 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1665d64d-c786-46c2-9ea8-1103d37d3b88 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:10.068745 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a41da0-a908-4b13-a623-89598c2bf508 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:10.068943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:10.100868 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c4d591-1b9a-4676-9a4a-09859007283d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:10.101467 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c4d591-1b9a-4676-9a4a-09859007283d None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:10.101760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c4d591-1b9a-4676-9a4a-09859007283d None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:10.102048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c4d591-1b9a-4676-9a4a-09859007283d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:10.102775 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c4d591-1b9a-4676-9a4a-09859007283d None admin] RBAC: Adding query filter params (role.id=20d83a8838004f60a32f8427577c7c35, scope.project.id=c830a3bfece6456486f4b094076b5364, user.id=a1586ee841cd44658e4d5feb25de7940) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:10.103082 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c4d591-1b9a-4676-9a4a-09859007283d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:10.104538 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c4d591-1b9a-4676-9a4a-09859007283d None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:10.117776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 260/525] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:07:09 2022] GET /identity/v3/role_assignments?user.id=a1586ee841cd44658e4d5feb25de7940&scope.project.id=c830a3bfece6456486f4b094076b5364&role.id=20d83a8838004f60a32f8427577c7c35 => generated 259 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:10.117776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c4d591-1b9a-4676-9a4a-09859007283d None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:10.187115 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a41da0-a908-4b13-a623-89598c2bf508 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:10.237902 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6194a88c-294a-4e61-9430-eb5e75617423 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:10.238541 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6194a88c-294a-4e61-9430-eb5e75617423 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:10.238818 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6194a88c-294a-4e61-9430-eb5e75617423 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:10.239087 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6194a88c-294a-4e61-9430-eb5e75617423 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:10.247560 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6194a88c-294a-4e61-9430-eb5e75617423 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:10.248508 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c4d591-1b9a-4676-9a4a-09859007283d None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:10.249026 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6194a88c-294a-4e61-9430-eb5e75617423 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:10.263269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-6194a88c-294a-4e61-9430-eb5e75617423 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 27 04:07:10.263269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:10.263269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:10.263269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:10.263269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:10.263269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:10.263269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:10.263269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:10.263269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:10.263269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:10.263269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:10.263269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:10.263269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:10.263269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:10.263269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:10.263269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:10.264420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:10.264420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:10.264420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:10.264420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:10.264420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:10.264420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:10.264420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:10.264420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:10.264420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:10.264420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:10.264420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:10.264420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:10.264420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:10.264420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:10.264420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:10.264420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:10.264420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:10.264420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:10.265550 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:10.265550 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:10.265550 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:10.265550 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:10.265550 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:10.265550 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:10.265550 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:10.265550 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:10.265550 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 27 04:07:10.265550 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:10.269210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 266/526] 10.222.0.26 () {56 vars in 1102 bytes} [Tue Sep 27 04:07:10 2022] GET /identity/v3/users/neutron => generated 85 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:10.270161 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6194a88c-294a-4e61-9430-eb5e75617423 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:10.300919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c38cad-c7c1-4886-a198-382083589a1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:10.302367 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c38cad-c7c1-4886-a198-382083589a1d None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:10.302740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c38cad-c7c1-4886-a198-382083589a1d None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:10.303103 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c38cad-c7c1-4886-a198-382083589a1d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:10.303837 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:10.314074 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c38cad-c7c1-4886-a198-382083589a1d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:10.316598 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c38cad-c7c1-4886-a198-382083589a1d None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:10.331848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-16c38cad-c7c1-4886-a198-382083589a1d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:10.331848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:10.331848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:10.331848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:10.331848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:10.331848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:10.331848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:10.331848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:10.331848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:10.331848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:10.331848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:10.331848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:10.331848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:10.331848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:10.331848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:10.332718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:10.332718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:10.332718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:10.332718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:10.332718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:10.332718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:10.332718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:10.332718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:10.332718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:10.332718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:10.332718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:10.332718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:10.332718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:10.332718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:10.332718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:10.332718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:10.332718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:10.333597 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:10.333597 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:10.333597 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:10.333597 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:10.333597 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:10.333597 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:10.333597 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:10.333597 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:10.338087 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 261/527] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:10.338927 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c38cad-c7c1-4886-a198-382083589a1d None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:10.373127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:10.397371 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6194a88c-294a-4e61-9430-eb5e75617423 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:10.444474 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2aad44f-e53d-49d3-bf2d-43b594ccbcca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:10.445112 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2aad44f-e53d-49d3-bf2d-43b594ccbcca None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:10.445396 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2aad44f-e53d-49d3-bf2d-43b594ccbcca None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:10.445671 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2aad44f-e53d-49d3-bf2d-43b594ccbcca None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:10.458473 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2aad44f-e53d-49d3-bf2d-43b594ccbcca None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:10.459883 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c38cad-c7c1-4886-a198-382083589a1d None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:10.460007 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2aad44f-e53d-49d3-bf2d-43b594ccbcca None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:10.477488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-d2aad44f-e53d-49d3-bf2d-43b594ccbcca None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:10.477488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:10.477488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:10.477488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:10.477488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:10.477488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:10.477488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:10.477488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:10.477488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:10.477488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:10.477488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:10.477488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:10.477488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:10.477488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:10.477488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:10.477488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:10.478495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:10.478495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:10.478495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:10.478495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:10.478495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:10.478495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:10.478495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:10.478495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:10.478495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:10.478495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:10.478495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:10.478495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:10.478495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:10.478495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:10.478495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:10.478495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:10.478495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:10.479334 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:10.479334 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:10.479334 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:10.479334 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 267/528] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:10.480137 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2aad44f-e53d-49d3-bf2d-43b594ccbcca None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:10.499972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c3f3f4-3b3c-4a84-9ed5-9e37fdb9537d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:10.500775 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c3f3f4-3b3c-4a84-9ed5-9e37fdb9537d None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:10.501061 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c3f3f4-3b3c-4a84-9ed5-9e37fdb9537d None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:10.501381 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c3f3f4-3b3c-4a84-9ed5-9e37fdb9537d None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:10.502147 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c3f3f4-3b3c-4a84-9ed5-9e37fdb9537d None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:10.502455 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c3f3f4-3b3c-4a84-9ed5-9e37fdb9537d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:10.504281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c3f3f4-3b3c-4a84-9ed5-9e37fdb9537d None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:10.523987 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 262/529] 10.222.0.26 () {56 vars in 1139 bytes} [Tue Sep 27 04:07:10 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:10.524867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c3f3f4-3b3c-4a84-9ed5-9e37fdb9537d None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:10.575254 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:10.602066 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2aad44f-e53d-49d3-bf2d-43b594ccbcca None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:10.622498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:10.641441 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c3f3f4-3b3c-4a84-9ed5-9e37fdb9537d None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:10.648670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ce7063-0228-4906-8f94-7e2797ff2475 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:10.649286 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ce7063-0228-4906-8f94-7e2797ff2475 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:10.649570 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ce7063-0228-4906-8f94-7e2797ff2475 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:10.649944 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ce7063-0228-4906-8f94-7e2797ff2475 None admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:10.650649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ce7063-0228-4906-8f94-7e2797ff2475 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:10.651008 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ce7063-0228-4906-8f94-7e2797ff2475 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:10.652438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ce7063-0228-4906-8f94-7e2797ff2475 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:10.672864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 268/530] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:10.673710 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ce7063-0228-4906-8f94-7e2797ff2475 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:10.685687 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce516aab-1910-4194-a40b-37beff3a8011 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:10.686319 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce516aab-1910-4194-a40b-37beff3a8011 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:10.686597 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce516aab-1910-4194-a40b-37beff3a8011 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:10.686973 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce516aab-1910-4194-a40b-37beff3a8011 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:10.687621 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce516aab-1910-4194-a40b-37beff3a8011 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:10.687922 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce516aab-1910-4194-a40b-37beff3a8011 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:10.689470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce516aab-1910-4194-a40b-37beff3a8011 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:10.702319 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 263/531] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:10.703147 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce516aab-1910-4194-a40b-37beff3a8011 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:10.801362 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ce7063-0228-4906-8f94-7e2797ff2475 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:10.826492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce516aab-1910-4194-a40b-37beff3a8011 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:10.843068 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba3fcac-314d-4757-a8c4-8d95af2be3e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:10.843677 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba3fcac-314d-4757-a8c4-8d95af2be3e6 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:10.844204 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba3fcac-314d-4757-a8c4-8d95af2be3e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:10.844479 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba3fcac-314d-4757-a8c4-8d95af2be3e6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:10.855245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:10.865396 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14be5527-83f6-4847-b97a-d56faa086e22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:10.866037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14be5527-83f6-4847-b97a-d56faa086e22 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:10.866362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14be5527-83f6-4847-b97a-d56faa086e22 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:10.866638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14be5527-83f6-4847-b97a-d56faa086e22 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:10.867466 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14be5527-83f6-4847-b97a-d56faa086e22 None admin] RBAC: Adding query filter params (role.id=20d83a8838004f60a32f8427577c7c35, scope.project.id=c830a3bfece6456486f4b094076b5364, user.id=8dcc53332e6b462baf349a72b19d2b3b) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:10.867826 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14be5527-83f6-4847-b97a-d56faa086e22 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:10.869363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14be5527-83f6-4847-b97a-d56faa086e22 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:10.875400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 264/532] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:07:10 2022] GET /identity/v3/role_assignments?user.id=8dcc53332e6b462baf349a72b19d2b3b&scope.project.id=c830a3bfece6456486f4b094076b5364&role.id=20d83a8838004f60a32f8427577c7c35 => generated 259 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:10.876109 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14be5527-83f6-4847-b97a-d56faa086e22 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:10.914167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:10.964067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba3fcac-314d-4757-a8c4-8d95af2be3e6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:10.965626 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba3fcac-314d-4757-a8c4-8d95af2be3e6 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:11.087336 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 269/533] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:07:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:07:11.098192 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba3fcac-314d-4757-a8c4-8d95af2be3e6 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:11.132856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14be5527-83f6-4847-b97a-d56faa086e22 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:11.139464 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:11.213483 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba3fcac-314d-4757-a8c4-8d95af2be3e6 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:11.262906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b6ee1d-30d0-44d9-9b37-8c9db693369f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:11.264341 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b6ee1d-30d0-44d9-9b37-8c9db693369f None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:11.264917 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b6ee1d-30d0-44d9-9b37-8c9db693369f None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:11.265408 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b6ee1d-30d0-44d9-9b37-8c9db693369f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:11.279123 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038b0e9b-abe2-4986-a886-e0e2bbf1a252 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:11.279123 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038b0e9b-abe2-4986-a886-e0e2bbf1a252 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:11.279123 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038b0e9b-abe2-4986-a886-e0e2bbf1a252 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:11.279413 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038b0e9b-abe2-4986-a886-e0e2bbf1a252 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:11.293109 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b6ee1d-30d0-44d9-9b37-8c9db693369f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:11.294642 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b6ee1d-30d0-44d9-9b37-8c9db693369f None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:11.299444 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038b0e9b-abe2-4986-a886-e0e2bbf1a252 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:11.300884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038b0e9b-abe2-4986-a886-e0e2bbf1a252 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:11.312758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-c1b6ee1d-30d0-44d9-9b37-8c9db693369f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:11.312758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:11.312758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:11.312758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:11.312758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:11.312758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:11.312758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:11.312758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:11.312758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:11.312758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:11.312758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:11.312758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:11.312758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:11.312758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:11.312758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:11.313633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:11.313633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:11.313633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:11.313633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:11.313633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:11.313633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:11.313633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:11.313633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:11.313633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:11.313633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:11.313633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:11.313633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:11.313633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:11.313633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:11.313633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:11.313633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:11.313633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:11.314505 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:11.314505 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:11.314505 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:11.314505 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:11.314505 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:11.314505 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:11.314505 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:11.314505 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:11.314505 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 265/534] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 438 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:11.320575 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b6ee1d-30d0-44d9-9b37-8c9db693369f None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:11.332067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-038b0e9b-abe2-4986-a886-e0e2bbf1a252 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:11.332067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:11.332067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:11.332067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:11.332067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:11.332067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:11.332067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:11.332067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:11.332067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:11.332067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:11.332067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:11.332067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:11.332067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:11.332067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:11.332067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:11.332067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:11.332969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:11.332969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:11.332969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:11.332969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:11.332969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:11.332969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:11.332969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:11.332969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:11.332969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:11.332969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:11.332969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:11.332969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:11.332969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:11.332969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:11.332969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:11.332969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:11.332969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:11.334417 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:11.334417 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:11.334417 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:11.345857 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 270/535] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:11.346730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038b0e9b-abe2-4986-a886-e0e2bbf1a252 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:11.405402 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:11.439043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b6ee1d-30d0-44d9-9b37-8c9db693369f None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:11.474490 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada8054b-d269-46a5-b3dc-e6272414100c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:11.475474 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada8054b-d269-46a5-b3dc-e6272414100c None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:11.475946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada8054b-d269-46a5-b3dc-e6272414100c None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:11.476427 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada8054b-d269-46a5-b3dc-e6272414100c None admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:11.477310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada8054b-d269-46a5-b3dc-e6272414100c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:11.477838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada8054b-d269-46a5-b3dc-e6272414100c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:11.479532 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada8054b-d269-46a5-b3dc-e6272414100c None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:11.492362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 266/536] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:11.505407 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada8054b-d269-46a5-b3dc-e6272414100c None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:11.690283 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada8054b-d269-46a5-b3dc-e6272414100c None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:11.702090 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:11.712911 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038b0e9b-abe2-4986-a886-e0e2bbf1a252 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:11.738985 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:11.763884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6dd2abe-e624-40bb-bb8c-3b519170076d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:11.765433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dd2abe-e624-40bb-bb8c-3b519170076d None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:11.769547 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dd2abe-e624-40bb-bb8c-3b519170076d None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:11.770367 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dd2abe-e624-40bb-bb8c-3b519170076d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:11.771685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6dd2abe-e624-40bb-bb8c-3b519170076d None admin] RBAC: Adding query filter params (role.id=20d83a8838004f60a32f8427577c7c35, scope.project.id=c830a3bfece6456486f4b094076b5364, user.id=3970310fc7e34704b686e8d3bff3fdb2) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:11.774638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6dd2abe-e624-40bb-bb8c-3b519170076d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:11.778541 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6dd2abe-e624-40bb-bb8c-3b519170076d None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:11.804261 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 267/537] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:07:11 2022] GET /identity/v3/role_assignments?user.id=3970310fc7e34704b686e8d3bff3fdb2&scope.project.id=c830a3bfece6456486f4b094076b5364&role.id=20d83a8838004f60a32f8427577c7c35 => generated 259 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:07:11.851804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b51a89-308a-414d-9a62-bf4670fced2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:11.852845 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b51a89-308a-414d-9a62-bf4670fced2b None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:11.853307 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b51a89-308a-414d-9a62-bf4670fced2b None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:11.853772 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b51a89-308a-414d-9a62-bf4670fced2b None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:11.854625 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b51a89-308a-414d-9a62-bf4670fced2b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:11.855380 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b51a89-308a-414d-9a62-bf4670fced2b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:11.862205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b51a89-308a-414d-9a62-bf4670fced2b None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:11.921903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6dd2abe-e624-40bb-bb8c-3b519170076d None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:11.922733 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 271/538] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 576 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:12.014315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03e0a71-45de-4002-a18e-64ae07a15621 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:12.015479 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03e0a71-45de-4002-a18e-64ae07a15621 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:12.015998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03e0a71-45de-4002-a18e-64ae07a15621 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:12.018088 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 272/539] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:12 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:07:12.023882 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1c7ce2-2da8-44ef-830a-0e93acbba835 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:12.024156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1c7ce2-2da8-44ef-830a-0e93acbba835 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:12.024430 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1c7ce2-2da8-44ef-830a-0e93acbba835 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:12.047377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-2a1c7ce2-2da8-44ef-830a-0e93acbba835 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:12.048597 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-2a1c7ce2-2da8-44ef-830a-0e93acbba835 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:12.156305 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 273/540] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:07:12.163583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6dd2abe-e624-40bb-bb8c-3b519170076d None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:12.206545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426d1465-d679-4d4b-a6de-5a046dd6fa38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:12.207172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426d1465-d679-4d4b-a6de-5a046dd6fa38 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:12.207445 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426d1465-d679-4d4b-a6de-5a046dd6fa38 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:12.207712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426d1465-d679-4d4b-a6de-5a046dd6fa38 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:12.220636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426d1465-d679-4d4b-a6de-5a046dd6fa38 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:12.222195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426d1465-d679-4d4b-a6de-5a046dd6fa38 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:12.239307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-426d1465-d679-4d4b-a6de-5a046dd6fa38 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:12.239307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:12.239307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:12.239307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:12.239307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:12.239307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:12.239307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:12.239307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:12.239307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:12.239307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:12.239307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:12.239307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:12.239307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:12.239307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:12.239307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:12.240656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:12.240656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:12.240656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:12.240656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:12.240656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:12.240656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:12.240656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:12.240656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:12.240656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:12.240656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:12.240656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:12.240656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:12.240656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:12.240656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:12.240656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:12.240656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:12.240656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:12.241491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:12.241491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:12.241491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:12.241491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:12.241491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:12.241491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:12.241491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:12.241491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:12.241491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 268/541] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 320 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:12.247967 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1c7ce2-2da8-44ef-830a-0e93acbba835 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:12.341857 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:12.363146 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1c7ce2-2da8-44ef-830a-0e93acbba835 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:12.409349 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fc9d37-c56b-4580-b749-0d21111483a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:12.410057 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fc9d37-c56b-4580-b749-0d21111483a6 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:12.410337 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fc9d37-c56b-4580-b749-0d21111483a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:12.410603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fc9d37-c56b-4580-b749-0d21111483a6 None admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:12.411936 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34fc9d37-c56b-4580-b749-0d21111483a6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:12.412242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34fc9d37-c56b-4580-b749-0d21111483a6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:12.413674 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34fc9d37-c56b-4580-b749-0d21111483a6 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:12.426439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 274/542] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:12.432456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426d1465-d679-4d4b-a6de-5a046dd6fa38 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:12.507494 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:12.562481 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426d1465-d679-4d4b-a6de-5a046dd6fa38 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:12.601706 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fc9d37-c56b-4580-b749-0d21111483a6 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:12.613350 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4587550b-6e98-47eb-bdd5-76526c0c3ff6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:12.614477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4587550b-6e98-47eb-bdd5-76526c0c3ff6 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:12.614980 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4587550b-6e98-47eb-bdd5-76526c0c3ff6 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:12.615448 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4587550b-6e98-47eb-bdd5-76526c0c3ff6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:12.616408 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4587550b-6e98-47eb-bdd5-76526c0c3ff6 None admin] RBAC: Adding query filter params (role.id=20d83a8838004f60a32f8427577c7c35, scope.project.id=c830a3bfece6456486f4b094076b5364, user.id=cf29a8da103b411fbc79004eb371c05d) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:12.616933 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4587550b-6e98-47eb-bdd5-76526c0c3ff6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:12.618629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4587550b-6e98-47eb-bdd5-76526c0c3ff6 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:12.625066 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 269/543] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:07:12 2022] GET /identity/v3/role_assignments?user.id=cf29a8da103b411fbc79004eb371c05d&scope.project.id=c830a3bfece6456486f4b094076b5364&role.id=20d83a8838004f60a32f8427577c7c35 => generated 259 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:12.728099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b97fe3-aaf5-4d99-821b-3255c5e93ed3 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:12.728746 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b97fe3-aaf5-4d99-821b-3255c5e93ed3 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:12.729214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b97fe3-aaf5-4d99-821b-3255c5e93ed3 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:12.731008 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 270/544] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:07:12.735962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f601f9c-e28a-44ec-9573-c9521ad79856 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:12.736351 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fc9d37-c56b-4580-b749-0d21111483a6 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:12.736783 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f601f9c-e28a-44ec-9573-c9521ad79856 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:12.737256 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f601f9c-e28a-44ec-9573-c9521ad79856 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:12.774466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ba3595-5ecc-461e-ae1e-48c59c731384 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:12.775100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ba3595-5ecc-461e-ae1e-48c59c731384 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:12.775377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ba3595-5ecc-461e-ae1e-48c59c731384 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:12.775646 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ba3595-5ecc-461e-ae1e-48c59c731384 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:12.777370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-9f601f9c-e28a-44ec-9573-c9521ad79856 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:12.778797 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-9f601f9c-e28a-44ec-9573-c9521ad79856 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:12.783987 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:12.788291 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ba3595-5ecc-461e-ae1e-48c59c731384 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:12.795332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-c5ba3595-5ecc-461e-ae1e-48c59c731384 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:12.795332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:12.795332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:12.795332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:12.795332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:12.795332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:12.795332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:12.795332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:12.795332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:12.795332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:12.795332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:12.795332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:12.795332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:12.795332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:12.795332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:12.795332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:12.796309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:12.796309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:12.796309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:12.796309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:12.796309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:12.796309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:12.796309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:12.796309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:12.796309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:12.796309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:12.796309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:12.796309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:12.796309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:12.796309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:12.796309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:12.796309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:12.797094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:12.797094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:12.797094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:12.797094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:12.797094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:12.797094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:12.797094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:12.797094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:12.797094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:12.797094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:12.797094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:12.797094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:12.799818 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 275/545] 10.222.0.26 () {56 vars in 1102 bytes} [Tue Sep 27 04:07:12 2022] GET /identity/v3/roles/service => generated 85 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:12.805434 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ba3595-5ecc-461e-ae1e-48c59c731384 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:12.922936 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ba3595-5ecc-461e-ae1e-48c59c731384 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:12.950840 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 271/546] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:07:12.962481 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db57f9df-1cf0-4b5f-beff-17206cd217ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:12.963378 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db57f9df-1cf0-4b5f-beff-17206cd217ba None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:12.963828 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db57f9df-1cf0-4b5f-beff-17206cd217ba None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:12.964263 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db57f9df-1cf0-4b5f-beff-17206cd217ba None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:12.965094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db57f9df-1cf0-4b5f-beff-17206cd217ba None admin] RBAC: Adding query filter params (name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:12.965583 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db57f9df-1cf0-4b5f-beff-17206cd217ba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:12.967129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db57f9df-1cf0-4b5f-beff-17206cd217ba None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:12.976026 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 276/547] 10.222.0.26 () {56 vars in 1103 bytes} [Tue Sep 27 04:07:12 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:12.981241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f601f9c-e28a-44ec-9573-c9521ad79856 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:13.034096 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:13.090004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f601f9c-e28a-44ec-9573-c9521ad79856 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:13.125371 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16800e3c-f2af-44a6-be47-49f46dd033d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:13.126302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16800e3c-f2af-44a6-be47-49f46dd033d5 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:13.126748 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16800e3c-f2af-44a6-be47-49f46dd033d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:13.127182 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16800e3c-f2af-44a6-be47-49f46dd033d5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:13.138359 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16800e3c-f2af-44a6-be47-49f46dd033d5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:13.139980 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16800e3c-f2af-44a6-be47-49f46dd033d5 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:13.161813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-16800e3c-f2af-44a6-be47-49f46dd033d5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:13.161813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:13.161813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:13.161813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:13.161813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:13.161813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:13.161813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:13.161813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:13.161813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:13.161813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:13.161813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:13.161813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:13.161813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:13.161813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:13.161813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:13.161813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:13.162767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:13.162767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:13.162767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:13.162767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:13.162767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:13.162767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:13.162767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:13.162767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:13.162767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:13.162767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:13.162767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:13.162767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:13.162767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:13.162767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:13.162767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:13.162767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:13.162767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:13.163578 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:13.163578 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:13.163578 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:13.166473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 272/548] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:07:13.171069 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db57f9df-1cf0-4b5f-beff-17206cd217ba None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:13.252320 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:13.285295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db57f9df-1cf0-4b5f-beff-17206cd217ba None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:13.320236 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90dec9ff-46eb-4cb0-80b2-73cb075e8475 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:13.321201 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90dec9ff-46eb-4cb0-80b2-73cb075e8475 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:13.321617 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90dec9ff-46eb-4cb0-80b2-73cb075e8475 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:13.322045 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90dec9ff-46eb-4cb0-80b2-73cb075e8475 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:13.322872 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90dec9ff-46eb-4cb0-80b2-73cb075e8475 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:13.323330 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90dec9ff-46eb-4cb0-80b2-73cb075e8475 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:13.324816 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90dec9ff-46eb-4cb0-80b2-73cb075e8475 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:13.338283 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 277/549] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:13.346284 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16800e3c-f2af-44a6-be47-49f46dd033d5 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:13.346443 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90dec9ff-46eb-4cb0-80b2-73cb075e8475 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:13.465358 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:13.469508 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16800e3c-f2af-44a6-be47-49f46dd033d5 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:13.514171 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b0dc67-dda6-49d8-aee7-478204da562b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:13.515176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b0dc67-dda6-49d8-aee7-478204da562b None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:13.515459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b0dc67-dda6-49d8-aee7-478204da562b None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:13.515728 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b0dc67-dda6-49d8-aee7-478204da562b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:13.528906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b0dc67-dda6-49d8-aee7-478204da562b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:13.536790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-c2b0dc67-dda6-49d8-aee7-478204da562b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:13.536790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:13.536790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:13.536790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:13.536790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:13.536790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:13.536790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:13.536790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:13.536790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:13.536790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:13.536790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:13.536790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:13.536790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:13.536790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:13.536790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:13.536790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:13.538222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:13.538222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:13.538222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:13.538222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:13.538222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:13.538222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:13.538222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:13.538222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:13.538222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:13.538222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:13.538222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:13.538222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:13.538222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:13.538222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:13.538222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:13.538222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:13.539179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:13.539179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:13.539179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:13.539179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:13.539179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:13.539179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:13.539179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:13.539179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:13.539179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:13.539179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:13.539179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:13.539179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:13.539179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 273/550] 10.222.0.26 () {56 vars in 1102 bytes} [Tue Sep 27 04:07:13 2022] GET /identity/v3/roles/service => generated 85 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:13.540315 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07db9438-3262-48b6-b1da-36a86fc5b454 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:13.540315 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07db9438-3262-48b6-b1da-36a86fc5b454 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:13.540594 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07db9438-3262-48b6-b1da-36a86fc5b454 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:13.544887 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 274/551] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:13.544887 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07db9438-3262-48b6-b1da-36a86fc5b454 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:13.545601 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90dec9ff-46eb-4cb0-80b2-73cb075e8475 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:13.574811 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:13.591903 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f623d26-84c4-464c-88d8-f4212d2fd618 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:13.593630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f623d26-84c4-464c-88d8-f4212d2fd618 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:13.594145 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f623d26-84c4-464c-88d8-f4212d2fd618 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:13.594657 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f623d26-84c4-464c-88d8-f4212d2fd618 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:13.605105 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f623d26-84c4-464c-88d8-f4212d2fd618 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:13.607339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f623d26-84c4-464c-88d8-f4212d2fd618 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:13.624136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-9f623d26-84c4-464c-88d8-f4212d2fd618 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 27 04:07:13.624136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:13.624136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:13.624136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:13.624136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:13.624136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:13.624136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:13.624136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:13.624136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:13.624136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:13.624136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:13.624136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:13.624136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:13.624136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:13.624136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:13.624136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:13.625050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:13.625050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:13.625050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:13.625050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:13.625050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:13.625050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:13.625050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:13.625050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:13.625050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:13.625050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:13.625050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:13.625050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:13.625050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:13.625050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:13.625050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:13.625050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:13.625050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:13.625050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:13.626173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:13.626173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:13.626173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:13.626173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:13.626173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:13.626173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:13.626173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:13.626173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:13.626173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 27 04:07:13.626173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:13.627660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 278/552] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:07:13 2022] GET /identity/v3/users/swift => generated 83 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:13.628637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22bff9d-51ea-48df-a664-cb3a03c31861 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:13.628918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22bff9d-51ea-48df-a664-cb3a03c31861 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:13.629192 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22bff9d-51ea-48df-a664-cb3a03c31861 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:13.646811 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-c22bff9d-51ea-48df-a664-cb3a03c31861 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:13.648018 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c22bff9d-51ea-48df-a664-cb3a03c31861 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:13.675335 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07db9438-3262-48b6-b1da-36a86fc5b454 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:13.719025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 279/553] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:07:13.719760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22bff9d-51ea-48df-a664-cb3a03c31861 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:13.724501 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9843be-1bc1-441e-86be-a72cfd993d90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:13.725434 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9843be-1bc1-441e-86be-a72cfd993d90 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:13.725945 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9843be-1bc1-441e-86be-a72cfd993d90 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:13.726409 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9843be-1bc1-441e-86be-a72cfd993d90 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:13.727324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9843be-1bc1-441e-86be-a72cfd993d90 None admin] RBAC: Adding query filter params (name=service) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:13.727841 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9843be-1bc1-441e-86be-a72cfd993d90 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:13.729925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9843be-1bc1-441e-86be-a72cfd993d90 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:13.743381 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 275/554] 10.222.0.26 () {56 vars in 1103 bytes} [Tue Sep 27 04:07:13 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:13.753038 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9843be-1bc1-441e-86be-a72cfd993d90 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:13.797453 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:13.888842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22bff9d-51ea-48df-a664-cb3a03c31861 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:13.912702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:13.934496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9843be-1bc1-441e-86be-a72cfd993d90 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:13.956583 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7ab425-87a3-43c0-85c2-aff290d0bd05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:13.957182 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7ab425-87a3-43c0-85c2-aff290d0bd05 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:13.957463 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7ab425-87a3-43c0-85c2-aff290d0bd05 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:13.957738 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7ab425-87a3-43c0-85c2-aff290d0bd05 None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:13.959016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7ab425-87a3-43c0-85c2-aff290d0bd05 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:13.959321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7ab425-87a3-43c0-85c2-aff290d0bd05 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:13.960746 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7ab425-87a3-43c0-85c2-aff290d0bd05 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:13.994091 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 280/555] 10.222.0.26 () {56 vars in 1135 bytes} [Tue Sep 27 04:07:13 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:13.997518 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db0bb48-3d3d-4f72-a3df-b3053d1e4cd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:13.998933 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7ab425-87a3-43c0-85c2-aff290d0bd05 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:14.001661 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db0bb48-3d3d-4f72-a3df-b3053d1e4cd8 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:14.001965 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db0bb48-3d3d-4f72-a3df-b3053d1e4cd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:14.002239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db0bb48-3d3d-4f72-a3df-b3053d1e4cd8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:14.035378 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db0bb48-3d3d-4f72-a3df-b3053d1e4cd8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:14.037761 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db0bb48-3d3d-4f72-a3df-b3053d1e4cd8 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:14.063177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-1db0bb48-3d3d-4f72-a3df-b3053d1e4cd8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:14.063177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:14.063177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:14.063177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:14.063177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:14.063177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:14.063177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:14.063177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:14.063177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:14.063177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:14.063177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:14.063177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:14.063177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:14.063177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:14.063177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:14.063177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:14.064089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:14.064089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:14.064089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:14.064089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:14.064089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:14.064089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:14.064089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:14.064089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:14.064089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:14.064089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:14.064089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:14.064089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:14.064089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:14.064089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:14.064089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:14.064089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:14.064089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:14.064961 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:14.064961 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:14.064961 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:14.065629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 276/556] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:14.077870 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db0bb48-3d3d-4f72-a3df-b3053d1e4cd8 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:14.162984 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7ab425-87a3-43c0-85c2-aff290d0bd05 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:14.181971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:14.204030 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c375c49-c301-4dc4-8b90-625169376fe6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:14.204723 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c375c49-c301-4dc4-8b90-625169376fe6 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:14.205026 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c375c49-c301-4dc4-8b90-625169376fe6 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:14.205302 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c375c49-c301-4dc4-8b90-625169376fe6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:14.226282 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db0bb48-3d3d-4f72-a3df-b3053d1e4cd8 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:14.247906 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:14.273266 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2770b21b-f3e2-4b68-bd58-f2eb54137dd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:14.274450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2770b21b-f3e2-4b68-bd58-f2eb54137dd4 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:14.274726 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2770b21b-f3e2-4b68-bd58-f2eb54137dd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:14.275004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2770b21b-f3e2-4b68-bd58-f2eb54137dd4 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:14.275653 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2770b21b-f3e2-4b68-bd58-f2eb54137dd4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:14.275953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2770b21b-f3e2-4b68-bd58-f2eb54137dd4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:14.277500 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2770b21b-f3e2-4b68-bd58-f2eb54137dd4 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:14.280811 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c375c49-c301-4dc4-8b90-625169376fe6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:14.282529 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c375c49-c301-4dc4-8b90-625169376fe6 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:14.293106 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 277/557] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:07:14.294311 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2770b21b-f3e2-4b68-bd58-f2eb54137dd4 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:14.396429 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 281/558] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:07:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:07:14.396429 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f656c0-017b-4ee2-aceb-a09f57d4294f None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:14.396429 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f656c0-017b-4ee2-aceb-a09f57d4294f None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:14.396429 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f656c0-017b-4ee2-aceb-a09f57d4294f None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:14.396429 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 282/559] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:14 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:14.396429 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f656c0-017b-4ee2-aceb-a09f57d4294f None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:14.413668 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2770b21b-f3e2-4b68-bd58-f2eb54137dd4 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:14.451915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a5e0ca-8443-496e-853c-4e0d919a4291 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:14.452532 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a5e0ca-8443-496e-853c-4e0d919a4291 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:14.452802 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a5e0ca-8443-496e-853c-4e0d919a4291 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:14.453067 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a5e0ca-8443-496e-853c-4e0d919a4291 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:14.454974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:14.463964 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a5e0ca-8443-496e-853c-4e0d919a4291 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:14.468790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-b0a5e0ca-8443-496e-853c-4e0d919a4291 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:14.468790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:14.468790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:14.468790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:14.468790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:14.468790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:14.468790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:14.468790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:14.468790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:14.468790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:14.468790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:14.468790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:14.468790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:14.468790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:14.468790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:14.468790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:14.469693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:14.469693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:14.469693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:14.469693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:14.469693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:14.469693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:14.469693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:14.469693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:14.469693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:14.469693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:14.469693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:14.469693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:14.469693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:14.469693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:14.469693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:14.469693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:14.470587 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:14.470587 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:14.470587 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:14.470587 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:14.470587 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:14.470587 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:14.470587 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:14.470587 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:14.470587 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:14.470587 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:14.470587 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:14.470587 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:14.470587 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 278/560] 10.222.0.26 () {56 vars in 1102 bytes} [Tue Sep 27 04:07:14 2022] GET /identity/v3/roles/service => generated 85 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:14.471287 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b127838-1c79-4088-8d05-655e1c7618f9 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:14.471867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b127838-1c79-4088-8d05-655e1c7618f9 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:14.471867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b127838-1c79-4088-8d05-655e1c7618f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:14.480993 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:14.490358 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-7b127838-1c79-4088-8d05-655e1c7618f9 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:14.491527 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-7b127838-1c79-4088-8d05-655e1c7618f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:14.515135 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f656c0-017b-4ee2-aceb-a09f57d4294f None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:14.551738 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daebac44-0e11-4099-b9c5-d63c6ff8a436 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:14.553205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daebac44-0e11-4099-b9c5-d63c6ff8a436 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:14.553485 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daebac44-0e11-4099-b9c5-d63c6ff8a436 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:14.553891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daebac44-0e11-4099-b9c5-d63c6ff8a436 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:14.569950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 279/561] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:07:14.569950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b127838-1c79-4088-8d05-655e1c7618f9 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:14.571769 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daebac44-0e11-4099-b9c5-d63c6ff8a436 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:14.573636 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daebac44-0e11-4099-b9c5-d63c6ff8a436 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:14.586468 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-daebac44-0e11-4099-b9c5-d63c6ff8a436 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 27 04:07:14.586468 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:14.586468 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:14.586468 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:14.586468 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:14.586468 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:14.586468 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:14.586468 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:14.586468 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:14.586468 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:14.586468 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:14.586468 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:14.586468 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:14.586468 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:14.586468 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:14.586468 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:14.587453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:14.587453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:14.587453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:14.587453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:14.587453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:14.587453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:14.587453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:14.587453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:14.587453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:14.587453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:14.587453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:14.587453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:14.587453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:14.587453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:14.587453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:14.587453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:14.587453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:14.587453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:14.588368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:14.588368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:14.588368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:14.588368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:14.588368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:14.588368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:14.588368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:14.588368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:14.588368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 27 04:07:14.588368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:14.592736 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 283/562] 10.222.0.26 () {56 vars in 1093 bytes} [Tue Sep 27 04:07:14 2022] GET /identity/v3/users/nova => generated 82 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:14.593629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daebac44-0e11-4099-b9c5-d63c6ff8a436 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:14.671239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b127838-1c79-4088-8d05-655e1c7618f9 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:14.696072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:14.709034 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a802fc5c-b83c-4d24-a8ff-70bb6af4c42d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:14.709759 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a802fc5c-b83c-4d24-a8ff-70bb6af4c42d None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:14.710104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a802fc5c-b83c-4d24-a8ff-70bb6af4c42d None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:14.710379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a802fc5c-b83c-4d24-a8ff-70bb6af4c42d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:14.724785 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a802fc5c-b83c-4d24-a8ff-70bb6af4c42d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:14.726723 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a802fc5c-b83c-4d24-a8ff-70bb6af4c42d None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:14.729979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daebac44-0e11-4099-b9c5-d63c6ff8a436 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:14.743834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-a802fc5c-b83c-4d24-a8ff-70bb6af4c42d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:14.743834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:14.743834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:14.743834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:14.743834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:14.743834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:14.743834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:14.743834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:14.743834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:14.743834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:14.743834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:14.743834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:14.743834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:14.743834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:14.743834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:14.743834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:14.744948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:14.744948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:14.744948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:14.744948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:14.744948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:14.744948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:14.744948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:14.744948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:14.744948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:14.744948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:14.744948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:14.744948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:14.744948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:14.744948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:14.744948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:14.744948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:14.744948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:14.745894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:14.745894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:14.745894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:14.745894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 280/563] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:14.746207 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a802fc5c-b83c-4d24-a8ff-70bb6af4c42d None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:14.766277 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:14.776171 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51cabd30-1ad9-41da-acd1-4fae7dc8a6b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:14.777201 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51cabd30-1ad9-41da-acd1-4fae7dc8a6b5 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:14.778056 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51cabd30-1ad9-41da-acd1-4fae7dc8a6b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:14.778343 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51cabd30-1ad9-41da-acd1-4fae7dc8a6b5 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:14.779019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51cabd30-1ad9-41da-acd1-4fae7dc8a6b5 None admin] RBAC: Adding query filter params (name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:14.779327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51cabd30-1ad9-41da-acd1-4fae7dc8a6b5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:14.780702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51cabd30-1ad9-41da-acd1-4fae7dc8a6b5 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:14.790081 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 284/564] 10.222.0.26 () {56 vars in 1103 bytes} [Tue Sep 27 04:07:14 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:14.791001 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51cabd30-1ad9-41da-acd1-4fae7dc8a6b5 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:14.864078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a802fc5c-b83c-4d24-a8ff-70bb6af4c42d None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:14.892037 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51cabd30-1ad9-41da-acd1-4fae7dc8a6b5 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:14.900087 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d1f447-df42-4a59-8be7-eaaeb3454099 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:14.900892 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d1f447-df42-4a59-8be7-eaaeb3454099 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:14.901363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d1f447-df42-4a59-8be7-eaaeb3454099 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:14.901849 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d1f447-df42-4a59-8be7-eaaeb3454099 None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:14.902634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d1f447-df42-4a59-8be7-eaaeb3454099 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:14.902933 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d1f447-df42-4a59-8be7-eaaeb3454099 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:14.904339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d1f447-df42-4a59-8be7-eaaeb3454099 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:14.923135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 281/565] 10.222.0.26 () {56 vars in 1133 bytes} [Tue Sep 27 04:07:14 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:14.923887 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d1f447-df42-4a59-8be7-eaaeb3454099 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:14.931105 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d319ea41-1174-435a-99c7-f41a34be76a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:14.931714 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d319ea41-1174-435a-99c7-f41a34be76a2 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:14.932003 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d319ea41-1174-435a-99c7-f41a34be76a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:14.932279 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d319ea41-1174-435a-99c7-f41a34be76a2 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:14.932930 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d319ea41-1174-435a-99c7-f41a34be76a2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:14.933231 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d319ea41-1174-435a-99c7-f41a34be76a2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:14.934630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d319ea41-1174-435a-99c7-f41a34be76a2 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:14.947073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 285/566] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:14.947886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d319ea41-1174-435a-99c7-f41a34be76a2 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:14.964086 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:15.012312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:15.031639 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d1f447-df42-4a59-8be7-eaaeb3454099 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:15.063875 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d319ea41-1174-435a-99c7-f41a34be76a2 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:15.068376 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da521df-a54e-4968-8e3c-419ed0709db0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:15.069012 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da521df-a54e-4968-8e3c-419ed0709db0 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:15.069293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da521df-a54e-4968-8e3c-419ed0709db0 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:15.069566 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da521df-a54e-4968-8e3c-419ed0709db0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:15.079354 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da521df-a54e-4968-8e3c-419ed0709db0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:15.080789 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da521df-a54e-4968-8e3c-419ed0709db0 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:15.095593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-8da521df-a54e-4968-8e3c-419ed0709db0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:15.095593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:15.095593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:15.095593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:15.095593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:15.095593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:15.095593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:15.095593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:15.095593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:15.095593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:15.095593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:15.095593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:15.095593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:15.095593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:15.095593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:15.095593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:15.096718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:15.096718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:15.096718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:15.096718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:15.096718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:15.096718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:15.096718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:15.096718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:15.096718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:15.096718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:15.096718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:15.096718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:15.096718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:15.096718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:15.096718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:15.096718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:15.096718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:15.097589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:15.097589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:15.097589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:15.097589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 282/567] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:15.097839 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da521df-a54e-4968-8e3c-419ed0709db0 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:15.101918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac321bb-4eb9-48e3-a62c-6ae8603ac020 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:15.102880 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac321bb-4eb9-48e3-a62c-6ae8603ac020 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:15.103163 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac321bb-4eb9-48e3-a62c-6ae8603ac020 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:15.103431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac321bb-4eb9-48e3-a62c-6ae8603ac020 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:15.114621 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac321bb-4eb9-48e3-a62c-6ae8603ac020 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:15.119886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-5ac321bb-4eb9-48e3-a62c-6ae8603ac020 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:15.119886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:15.119886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:15.119886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:15.119886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:15.119886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:15.119886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:15.119886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:15.119886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:15.119886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:15.119886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:15.119886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:15.119886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:15.119886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:15.119886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:15.119886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:15.120916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:15.120916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:15.120916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:15.120916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:15.120916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:15.120916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:15.120916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:15.120916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:15.120916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:15.120916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:15.120916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:15.120916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:15.120916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:15.120916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:15.120916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:15.120916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:15.122746 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:15.122746 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:15.122746 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:15.122746 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:15.122746 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:15.122746 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:15.122746 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:15.122746 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:15.122746 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:15.122746 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:15.122746 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:15.122746 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:15.122746 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 286/568] 10.222.0.26 () {56 vars in 1102 bytes} [Tue Sep 27 04:07:14 2022] GET /identity/v3/roles/service => generated 85 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:15.122746 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac321bb-4eb9-48e3-a62c-6ae8603ac020 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:15.196104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:15.212990 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da521df-a54e-4968-8e3c-419ed0709db0 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:15.234274 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac321bb-4eb9-48e3-a62c-6ae8603ac020 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:15.247316 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062a7a2f-8e7c-4e5a-8842-058441a86f42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:15.247889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:15.248004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062a7a2f-8e7c-4e5a-8842-058441a86f42 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:15.248194 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062a7a2f-8e7c-4e5a-8842-058441a86f42 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:15.248462 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062a7a2f-8e7c-4e5a-8842-058441a86f42 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:15.260077 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062a7a2f-8e7c-4e5a-8842-058441a86f42 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:15.261521 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062a7a2f-8e7c-4e5a-8842-058441a86f42 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:15.270270 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b57633-20ff-4306-b694-923beab6c306 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:15.271446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b57633-20ff-4306-b694-923beab6c306 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:15.271729 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b57633-20ff-4306-b694-923beab6c306 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:15.272004 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b57633-20ff-4306-b694-923beab6c306 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:15.276804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-062a7a2f-8e7c-4e5a-8842-058441a86f42 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:15.276804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:15.276804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:15.276804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:15.276804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:15.276804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:15.276804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:15.276804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:15.276804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:15.276804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:15.276804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:15.276804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:15.276804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:15.276804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:15.276804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:15.276804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:15.277682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:15.277682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:15.277682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:15.277682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:15.277682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:15.277682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:15.277682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:15.277682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:15.277682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:15.277682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:15.277682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:15.277682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:15.277682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:15.277682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:15.277682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:15.277682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:15.277682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:15.278592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:15.278592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:15.278592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:15.278592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 283/569] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:15.278919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062a7a2f-8e7c-4e5a-8842-058441a86f42 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:15.279908 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b57633-20ff-4306-b694-923beab6c306 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:15.281321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b57633-20ff-4306-b694-923beab6c306 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:15.292573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-b5b57633-20ff-4306-b694-923beab6c306 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:15.292573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:15.292573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:15.292573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:15.292573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:15.292573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:15.292573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:15.292573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:15.292573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:15.292573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:15.292573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:15.292573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:15.292573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:15.292573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:15.292573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:15.293649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:15.293649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:15.293649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:15.293649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:15.293649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:15.293649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:15.293649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:15.293649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:15.293649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:15.293649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:15.293649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:15.293649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:15.293649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:15.293649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:15.293649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:15.293649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:15.293649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:15.294635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:15.294635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:15.294635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:15.294635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:15.294635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:15.294635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:15.294635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:15.294635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:15.294635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 287/570] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:15.294635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b57633-20ff-4306-b694-923beab6c306 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:15.382644 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062a7a2f-8e7c-4e5a-8842-058441a86f42 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:15.399122 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b57633-20ff-4306-b694-923beab6c306 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:15.416107 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f511b5-2f90-4b70-9283-735d45aac933 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:15.416729 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f511b5-2f90-4b70-9283-735d45aac933 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:15.417007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f511b5-2f90-4b70-9283-735d45aac933 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:15.417276 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f511b5-2f90-4b70-9283-735d45aac933 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:15.417950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f511b5-2f90-4b70-9283-735d45aac933 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:15.418251 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f511b5-2f90-4b70-9283-735d45aac933 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:15.419610 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f511b5-2f90-4b70-9283-735d45aac933 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:15.425983 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:15.432464 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef2693b-7053-4f39-ac9a-563b213615ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:15.432768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 284/571] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:15.433976 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aec655c-f87e-41a0-8d1b-33f2814bfc62 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:15.434098 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef2693b-7053-4f39-ac9a-563b213615ed None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:15.434262 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aec655c-f87e-41a0-8d1b-33f2814bfc62 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:15.434383 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef2693b-7053-4f39-ac9a-563b213615ed None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:15.434549 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aec655c-f87e-41a0-8d1b-33f2814bfc62 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:15.434652 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef2693b-7053-4f39-ac9a-563b213615ed None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:15.435293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef2693b-7053-4f39-ac9a-563b213615ed None admin] RBAC: Adding query filter params (name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:15.435590 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef2693b-7053-4f39-ac9a-563b213615ed None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:15.437785 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 285/572] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:15.437785 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aec655c-f87e-41a0-8d1b-33f2814bfc62 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:15.437956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef2693b-7053-4f39-ac9a-563b213615ed None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:15.445980 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 288/573] 10.222.0.26 () {56 vars in 1103 bytes} [Tue Sep 27 04:07:15 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:15.446756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef2693b-7053-4f39-ac9a-563b213615ed None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:15.482618 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:15.536075 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aec655c-f87e-41a0-8d1b-33f2814bfc62 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:15.554537 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef2693b-7053-4f39-ac9a-563b213615ed None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:15.568772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db5b7e0-551c-4b0a-8330-20f8d08d9247 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:15.569376 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db5b7e0-551c-4b0a-8330-20f8d08d9247 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:15.569656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db5b7e0-551c-4b0a-8330-20f8d08d9247 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:15.569964 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db5b7e0-551c-4b0a-8330-20f8d08d9247 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:15.570604 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db5b7e0-551c-4b0a-8330-20f8d08d9247 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:15.570902 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db5b7e0-551c-4b0a-8330-20f8d08d9247 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:15.572261 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db5b7e0-551c-4b0a-8330-20f8d08d9247 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:15.588490 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 286/574] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:15.588490 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db5b7e0-551c-4b0a-8330-20f8d08d9247 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:15.588749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d6869a-c1d2-42c6-ae09-ad4141d934ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:15.588749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d6869a-c1d2-42c6-ae09-ad4141d934ae None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:15.588984 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d6869a-c1d2-42c6-ae09-ad4141d934ae None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:15.589261 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d6869a-c1d2-42c6-ae09-ad4141d934ae None admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:15.589971 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d6869a-c1d2-42c6-ae09-ad4141d934ae None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:15.590275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d6869a-c1d2-42c6-ae09-ad4141d934ae None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:15.591666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d6869a-c1d2-42c6-ae09-ad4141d934ae None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:15.604188 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 289/575] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:15.605325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4162ee1-5d89-4a22-855c-0c1e43ed0e71 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:15.605604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4162ee1-5d89-4a22-855c-0c1e43ed0e71 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:15.605905 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4162ee1-5d89-4a22-855c-0c1e43ed0e71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:15.622571 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-e4162ee1-5d89-4a22-855c-0c1e43ed0e71 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:15.623733 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-e4162ee1-5d89-4a22-855c-0c1e43ed0e71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:15.641448 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:15.682042 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db5b7e0-551c-4b0a-8330-20f8d08d9247 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:15.689987 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 290/576] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:07:15.690802 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4162ee1-5d89-4a22-855c-0c1e43ed0e71 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:15.716218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:15.717438 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d402d2-16c5-4cb4-ade4-35f023a47fff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:15.718005 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d402d2-16c5-4cb4-ade4-35f023a47fff None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:15.718275 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d402d2-16c5-4cb4-ade4-35f023a47fff None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:15.718617 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d402d2-16c5-4cb4-ade4-35f023a47fff None admin] PATH_INFO: `/v3/users/glance` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:15.726021 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d402d2-16c5-4cb4-ade4-35f023a47fff None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:15.727451 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d402d2-16c5-4cb4-ade4-35f023a47fff None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:15.739590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-b6d402d2-16c5-4cb4-ade4-35f023a47fff None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 27 04:07:15.739590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:15.739590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:15.739590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:15.739590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:15.739590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:15.739590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:15.739590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:15.739590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:15.739590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:15.739590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:15.739590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:15.739590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:15.739590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:15.739590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:15.739590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:15.740694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:15.740694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:15.740694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:15.740694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:15.740694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:15.740694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:15.740694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:15.740694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:15.740694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:15.740694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:15.740694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:15.740694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:15.740694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:15.740694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:15.740694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:15.740694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:15.740694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:15.740694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:15.741663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:15.741663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:15.741663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:15.741663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:15.741663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:15.741663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:15.741663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:15.741663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:15.741663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 27 04:07:15.741663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:15.741663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 287/577] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:07:15 2022] GET /identity/v3/users/glance => generated 84 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:15.741663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d402d2-16c5-4cb4-ade4-35f023a47fff None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:15.795943 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4162ee1-5d89-4a22-855c-0c1e43ed0e71 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:15.829421 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e8e9ce-4fbf-423a-a053-76e6a23865db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:15.830069 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e8e9ce-4fbf-423a-a053-76e6a23865db None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:15.830349 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e8e9ce-4fbf-423a-a053-76e6a23865db None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:15.830616 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e8e9ce-4fbf-423a-a053-76e6a23865db None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:15.841165 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e8e9ce-4fbf-423a-a053-76e6a23865db None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:15.842617 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e8e9ce-4fbf-423a-a053-76e6a23865db None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:15.846733 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d402d2-16c5-4cb4-ade4-35f023a47fff None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:15.859148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-75e8e9ce-4fbf-423a-a053-76e6a23865db None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:15.859148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:15.859148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:15.859148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:15.859148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:15.859148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:15.859148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:15.859148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:15.859148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:15.859148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:15.859148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:15.859148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:15.859148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:15.859148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:15.859148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:15.859148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:15.860289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:15.860289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:15.860289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:15.860289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:15.860289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:15.860289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:15.860289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:15.860289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:15.860289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:15.860289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:15.860289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:15.860289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:15.860289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:15.860289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:15.860289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:15.860289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:15.860289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:15.861370 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:15.861370 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:15.861370 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:15.864987 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 291/578] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:15.865824 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e8e9ce-4fbf-423a-a053-76e6a23865db None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:15.867749 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:15.886094 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405f385e-71a6-4e25-85d3-7fb1c8e2ec1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:15.886792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405f385e-71a6-4e25-85d3-7fb1c8e2ec1a None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:15.887134 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405f385e-71a6-4e25-85d3-7fb1c8e2ec1a None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:15.887472 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405f385e-71a6-4e25-85d3-7fb1c8e2ec1a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:15.894224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405f385e-71a6-4e25-85d3-7fb1c8e2ec1a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:15.896301 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405f385e-71a6-4e25-85d3-7fb1c8e2ec1a None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:15.908658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-405f385e-71a6-4e25-85d3-7fb1c8e2ec1a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:15.908658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:15.908658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:15.908658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:15.908658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:15.908658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:15.908658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:15.908658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:15.908658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:15.908658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:15.908658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:15.908658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:15.908658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:15.908658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:15.908658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:15.909539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:15.909539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:15.909539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:15.909539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:15.909539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:15.909539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:15.909539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:15.909539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:15.909539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:15.909539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:15.909539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:15.909539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:15.909539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:15.909539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:15.909539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:15.909539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:15.909539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:15.910479 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:15.910479 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:15.910479 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:15.910479 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:15.910479 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:15.910479 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:15.910479 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:15.910479 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:15.912390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 288/579] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:15.913273 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405f385e-71a6-4e25-85d3-7fb1c8e2ec1a None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:15.954939 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:15.978847 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e8e9ce-4fbf-423a-a053-76e6a23865db None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:16.010143 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405f385e-71a6-4e25-85d3-7fb1c8e2ec1a None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:16.011436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254c68e9-84fb-4f20-b973-2c40a1d041a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:16.012031 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254c68e9-84fb-4f20-b973-2c40a1d041a4 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:16.012940 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254c68e9-84fb-4f20-b973-2c40a1d041a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:16.012940 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254c68e9-84fb-4f20-b973-2c40a1d041a4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:16.013295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254c68e9-84fb-4f20-b973-2c40a1d041a4 None admin] RBAC: Adding query filter params (role.id=20d83a8838004f60a32f8427577c7c35, scope.project.id=c830a3bfece6456486f4b094076b5364, user.id=181dd5dc720c4a97860852062b7b19c3) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:16.013603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254c68e9-84fb-4f20-b973-2c40a1d041a4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:16.015065 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254c68e9-84fb-4f20-b973-2c40a1d041a4 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:16.021176 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 292/580] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:07:15 2022] GET /identity/v3/role_assignments?user.id=181dd5dc720c4a97860852062b7b19c3&scope.project.id=c830a3bfece6456486f4b094076b5364&role.id=20d83a8838004f60a32f8427577c7c35 => generated 259 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:16.021982 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254c68e9-84fb-4f20-b973-2c40a1d041a4 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:16.046968 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f62143f-a96e-4e3f-98aa-569eac59b048 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:16.047890 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f62143f-a96e-4e3f-98aa-569eac59b048 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:16.048267 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f62143f-a96e-4e3f-98aa-569eac59b048 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:16.048641 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f62143f-a96e-4e3f-98aa-569eac59b048 None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:16.049507 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f62143f-a96e-4e3f-98aa-569eac59b048 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:16.049964 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f62143f-a96e-4e3f-98aa-569eac59b048 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:16.051606 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f62143f-a96e-4e3f-98aa-569eac59b048 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:16.072571 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 289/581] 10.222.0.26 () {56 vars in 1137 bytes} [Tue Sep 27 04:07:15 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:16.073455 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f62143f-a96e-4e3f-98aa-569eac59b048 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:16.116408 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:16.148240 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254c68e9-84fb-4f20-b973-2c40a1d041a4 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:16.185940 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb00b18-97aa-4792-bfb7-83319ca1492f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:16.186548 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb00b18-97aa-4792-bfb7-83319ca1492f None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:16.186835 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb00b18-97aa-4792-bfb7-83319ca1492f None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:16.187110 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb00b18-97aa-4792-bfb7-83319ca1492f None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:16.187207 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f62143f-a96e-4e3f-98aa-569eac59b048 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:16.187883 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb00b18-97aa-4792-bfb7-83319ca1492f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:16.188065 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb00b18-97aa-4792-bfb7-83319ca1492f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:16.190593 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb00b18-97aa-4792-bfb7-83319ca1492f None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:16.201774 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:16.209240 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 293/582] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:16.210425 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb00b18-97aa-4792-bfb7-83319ca1492f None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:16.226812 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd3e72c-8fdf-41c8-bca3-72793730ab18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:16.227451 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd3e72c-8fdf-41c8-bca3-72793730ab18 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:16.227677 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd3e72c-8fdf-41c8-bca3-72793730ab18 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:16.227944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd3e72c-8fdf-41c8-bca3-72793730ab18 None admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:16.228616 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd3e72c-8fdf-41c8-bca3-72793730ab18 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:16.228970 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd3e72c-8fdf-41c8-bca3-72793730ab18 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:16.230369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd3e72c-8fdf-41c8-bca3-72793730ab18 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:16.243238 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 290/583] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:16.244285 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd3e72c-8fdf-41c8-bca3-72793730ab18 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:16.331641 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb00b18-97aa-4792-bfb7-83319ca1492f None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:16.341641 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:16.348676 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd3e72c-8fdf-41c8-bca3-72793730ab18 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:16.368768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31dd9269-f0fe-4dd4-bfff-308760674ecb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:16.369394 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31dd9269-f0fe-4dd4-bfff-308760674ecb None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:16.370117 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31dd9269-f0fe-4dd4-bfff-308760674ecb None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:16.370397 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31dd9269-f0fe-4dd4-bfff-308760674ecb None admin] PATH_INFO: `/v3/roles/service` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:16.381865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31dd9269-f0fe-4dd4-bfff-308760674ecb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:16.384486 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ebf700-f4ae-4d3d-b432-ac99d3823456 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:16.385105 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ebf700-f4ae-4d3d-b432-ac99d3823456 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:16.385379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ebf700-f4ae-4d3d-b432-ac99d3823456 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:16.385647 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ebf700-f4ae-4d3d-b432-ac99d3823456 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:16.386635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-31dd9269-f0fe-4dd4-bfff-308760674ecb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:16.386635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:16.386635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:16.386635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:16.386635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:16.386635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:16.386635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:16.386635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:16.386635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:16.386635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:16.386635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:16.386635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:16.386635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:16.386635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:16.386635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:16.386635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:16.387517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:16.387517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:16.387517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:16.387517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:16.387517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:16.387517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:16.387517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:16.387517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:16.387517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:16.387517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:16.387517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:16.387517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:16.387517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:16.387517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:16.387517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:16.387517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:16.388366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:16.388366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:16.388366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:16.388366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:16.388366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:16.388366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:16.388366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:16.388366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:16.388366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:16.388366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:16.388366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:16.388366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:16.388366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 294/584] 10.222.0.26 () {56 vars in 1102 bytes} [Tue Sep 27 04:07:16 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:16.389058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31dd9269-f0fe-4dd4-bfff-308760674ecb None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:16.402184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ebf700-f4ae-4d3d-b432-ac99d3823456 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:16.403628 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ebf700-f4ae-4d3d-b432-ac99d3823456 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:16.418369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-c9ebf700-f4ae-4d3d-b432-ac99d3823456 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:16.418369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:16.418369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:16.418369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:16.418369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:16.418369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:16.418369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:16.418369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:16.418369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:16.418369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:16.418369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:16.418369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:16.418369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:16.418369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:16.418369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:16.418369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:16.419324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:16.419324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:16.419324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:16.419324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:16.419324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:16.419324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:16.419324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:16.419324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:16.419324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:16.419324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:16.419324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:16.419324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:16.419324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:16.419324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:16.419324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:16.419324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:16.419324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:16.420208 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:16.420208 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:16.420208 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:16.420208 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 291/585] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:16.420208 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ebf700-f4ae-4d3d-b432-ac99d3823456 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:16.460740 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:16.512160 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31dd9269-f0fe-4dd4-bfff-308760674ecb None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:16.524805 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ebf700-f4ae-4d3d-b432-ac99d3823456 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:16.553671 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed00362-2320-42c0-80f3-bc39efa5eb8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:16.554429 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed00362-2320-42c0-80f3-bc39efa5eb8a None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:16.554706 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed00362-2320-42c0-80f3-bc39efa5eb8a None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:16.554973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed00362-2320-42c0-80f3-bc39efa5eb8a None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:16.559182 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15764a0-94b6-4fcd-a70c-9c6c8d7990e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:16.559822 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15764a0-94b6-4fcd-a70c-9c6c8d7990e3 None admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:16.560098 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15764a0-94b6-4fcd-a70c-9c6c8d7990e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:16.560366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15764a0-94b6-4fcd-a70c-9c6c8d7990e3 None admin] PATH_INFO: `/v3/projects/c830a3bfece6456486f4b094076b5364/users/a1586ee841cd44658e4d5feb25de7940/roles/20d83a8838004f60a32f8427577c7c35` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:16.562225 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ed00362-2320-42c0-80f3-bc39efa5eb8a None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:16.564795 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ed00362-2320-42c0-80f3-bc39efa5eb8a None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:16.574750 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d15764a0-94b6-4fcd-a70c-9c6c8d7990e3 None admin] RBAC: Authorizing `identity:create_grant(project_id=c830a3bfece6456486f4b094076b5364, user_id=a1586ee841cd44658e4d5feb25de7940, role_id=20d83a8838004f60a32f8427577c7c35)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:16.576227 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d15764a0-94b6-4fcd-a70c-9c6c8d7990e3 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:16.577229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-4ed00362-2320-42c0-80f3-bc39efa5eb8a None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 27 04:07:16.577229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:16.577229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:16.577229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:16.577229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:16.577229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:16.577229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:16.577229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:16.577229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:16.577229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:16.577229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:16.577229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:16.577229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:16.577229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:16.577229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:16.577229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:16.578376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:16.578376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:16.578376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:16.578376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:16.578376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:16.578376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:16.578376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:16.578376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:16.578376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:16.578376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:16.578376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:16.578376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:16.578376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:16.578376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:16.578376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:16.578376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:16.578376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:16.578376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:16.579329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:16.579329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:16.579329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:16.579329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:16.579329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:16.579329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:16.579329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:16.579329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:16.579329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 27 04:07:16.579329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:16.581884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 295/586] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:07:16 2022] GET /identity/v3/users/cinder => generated 84 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:16.582685 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed00362-2320-42c0-80f3-bc39efa5eb8a None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:16.589474 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:16.597931 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 292/587] 10.222.0.26 () {58 vars in 1439 bytes} [Tue Sep 27 04:07:16 2022] PUT /identity/v3/projects/c830a3bfece6456486f4b094076b5364/users/a1586ee841cd44658e4d5feb25de7940/roles/20d83a8838004f60a32f8427577c7c35 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:07:16.599423 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15764a0-94b6-4fcd-a70c-9c6c8d7990e3 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:16.702441 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed00362-2320-42c0-80f3-bc39efa5eb8a None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:16.725066 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:16.727769 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15764a0-94b6-4fcd-a70c-9c6c8d7990e3 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:16.745284 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6324cc-3338-4d12-8909-4330da054e59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:16.745953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6324cc-3338-4d12-8909-4330da054e59 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:16.746238 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6324cc-3338-4d12-8909-4330da054e59 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:16.746510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6324cc-3338-4d12-8909-4330da054e59 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:16.747207 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6324cc-3338-4d12-8909-4330da054e59 None admin] RBAC: Adding query filter params (name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:16.747511 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6324cc-3338-4d12-8909-4330da054e59 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:16.748910 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6324cc-3338-4d12-8909-4330da054e59 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:16.762920 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 296/588] 10.222.0.26 () {56 vars in 1103 bytes} [Tue Sep 27 04:07:16 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:16.762920 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6324cc-3338-4d12-8909-4330da054e59 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:16.777924 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea4d415-d6c6-4a36-ae76-0cd3bc153f87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:16.778908 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea4d415-d6c6-4a36-ae76-0cd3bc153f87 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:16.779425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea4d415-d6c6-4a36-ae76-0cd3bc153f87 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:16.779938 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea4d415-d6c6-4a36-ae76-0cd3bc153f87 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:16.781343 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea4d415-d6c6-4a36-ae76-0cd3bc153f87 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:16.781921 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea4d415-d6c6-4a36-ae76-0cd3bc153f87 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:16.783487 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea4d415-d6c6-4a36-ae76-0cd3bc153f87 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:16.797428 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 293/589] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:16.798640 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea4d415-d6c6-4a36-ae76-0cd3bc153f87 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:16.866517 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:16.874002 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6324cc-3338-4d12-8909-4330da054e59 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:16.913419 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea4d415-d6c6-4a36-ae76-0cd3bc153f87 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:16.919067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef56ec5-9f44-41a0-919e-172cf5fe0c9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:16.919674 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef56ec5-9f44-41a0-919e-172cf5fe0c9d None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:16.919953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef56ec5-9f44-41a0-919e-172cf5fe0c9d None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:16.920225 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef56ec5-9f44-41a0-919e-172cf5fe0c9d None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:16.921467 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef56ec5-9f44-41a0-919e-172cf5fe0c9d None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:16.921857 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef56ec5-9f44-41a0-919e-172cf5fe0c9d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:16.923737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef56ec5-9f44-41a0-919e-172cf5fe0c9d None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:16.944706 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 297/590] 10.222.0.26 () {56 vars in 1137 bytes} [Tue Sep 27 04:07:16 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:16.945549 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef56ec5-9f44-41a0-919e-172cf5fe0c9d None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:16.953948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f82b039-68b0-4bd0-969a-2941e53aeb35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:16.954873 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f82b039-68b0-4bd0-969a-2941e53aeb35 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:16.955388 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f82b039-68b0-4bd0-969a-2941e53aeb35 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:16.955907 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f82b039-68b0-4bd0-969a-2941e53aeb35 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:16.967633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f82b039-68b0-4bd0-969a-2941e53aeb35 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:16.969471 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f82b039-68b0-4bd0-969a-2941e53aeb35 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:16.984690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-1f82b039-68b0-4bd0-969a-2941e53aeb35 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:16.984690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:16.984690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:16.984690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:16.984690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:16.984690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:16.984690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:16.984690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:16.984690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:16.984690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:16.984690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:16.984690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:16.984690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:16.984690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:16.984690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:16.984690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:16.985614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:16.985614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:16.985614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:16.985614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:16.985614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:16.985614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:16.985614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:16.985614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:16.985614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:16.985614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:16.985614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:16.985614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:16.985614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:16.985614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:16.985614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:16.985614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:16.985614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:16.986820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:16.986820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:16.986820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:16.993213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:16.993711 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 294/591] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:16.994788 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f82b039-68b0-4bd0-969a-2941e53aeb35 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:17.051222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef56ec5-9f44-41a0-919e-172cf5fe0c9d None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:17.088260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea00d03-4c3b-4cf5-8cf9-71f412693892 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:17.088907 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea00d03-4c3b-4cf5-8cf9-71f412693892 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:17.089241 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea00d03-4c3b-4cf5-8cf9-71f412693892 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:17.089426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea00d03-4c3b-4cf5-8cf9-71f412693892 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:17.098357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea00d03-4c3b-4cf5-8cf9-71f412693892 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:17.100468 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea00d03-4c3b-4cf5-8cf9-71f412693892 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:17.102268 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f82b039-68b0-4bd0-969a-2941e53aeb35 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:17.113260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-0ea00d03-4c3b-4cf5-8cf9-71f412693892 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:17.113260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:17.113260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:17.113260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:17.113260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:17.113260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:17.113260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:17.113260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:17.113260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:17.113260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:17.113260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:17.113260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:17.113260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:17.113260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:17.113260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:17.114194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:17.114194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:17.114194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:17.114194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:17.114194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:17.114194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:17.114194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:17.114194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:17.114194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:17.114194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:17.114194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:17.114194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:17.114194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:17.114194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:17.114194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:17.114194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:17.114194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:17.115125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:17.115125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:17.115125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:17.115125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:17.115125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:17.115125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:17.115125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:17.115125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:17.115125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 298/592] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:17.115612 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea00d03-4c3b-4cf5-8cf9-71f412693892 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:17.116586 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:17.143154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ff4aff-cf47-408b-86e5-91b6c5228bc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:17.144175 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ff4aff-cf47-408b-86e5-91b6c5228bc4 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:17.144666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ff4aff-cf47-408b-86e5-91b6c5228bc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:17.145126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ff4aff-cf47-408b-86e5-91b6c5228bc4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:17.156911 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ff4aff-cf47-408b-86e5-91b6c5228bc4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:17.158664 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ff4aff-cf47-408b-86e5-91b6c5228bc4 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:17.174166 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-96ff4aff-cf47-408b-86e5-91b6c5228bc4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:17.174166 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:17.174166 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:17.174166 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:17.174166 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:17.174166 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:17.174166 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:17.174166 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:17.174166 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:17.174166 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:17.174166 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:17.174166 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:17.174166 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:17.174166 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:17.174166 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:17.174166 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:17.175123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:17.175123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:17.175123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:17.175123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:17.175123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:17.175123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:17.175123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:17.175123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:17.175123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:17.175123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:17.175123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:17.175123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:17.175123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:17.175123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:17.175123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:17.175123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:17.175123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:17.175996 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:17.175996 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:17.175996 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:17.182382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 295/593] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:17.183494 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ff4aff-cf47-408b-86e5-91b6c5228bc4 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:17.217643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:17.221162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea00d03-4c3b-4cf5-8cf9-71f412693892 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:17.262963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11379015-263e-4244-ae3d-5f3c06283388 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:17.263566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11379015-263e-4244-ae3d-5f3c06283388 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:17.263842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11379015-263e-4244-ae3d-5f3c06283388 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:17.264264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11379015-263e-4244-ae3d-5f3c06283388 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:17.264757 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11379015-263e-4244-ae3d-5f3c06283388 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:17.265054 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11379015-263e-4244-ae3d-5f3c06283388 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:17.266442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11379015-263e-4244-ae3d-5f3c06283388 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:17.279905 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 299/594] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:17.280853 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11379015-263e-4244-ae3d-5f3c06283388 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:17.311502 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ff4aff-cf47-408b-86e5-91b6c5228bc4 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:17.357765 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f64f0f-2d45-46e1-90ae-9fd73af406d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:17.358498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f64f0f-2d45-46e1-90ae-9fd73af406d2 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:17.358848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f64f0f-2d45-46e1-90ae-9fd73af406d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:17.359193 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f64f0f-2d45-46e1-90ae-9fd73af406d2 None admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:17.360507 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f64f0f-2d45-46e1-90ae-9fd73af406d2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:17.360887 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f64f0f-2d45-46e1-90ae-9fd73af406d2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:17.362867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f64f0f-2d45-46e1-90ae-9fd73af406d2 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:17.380972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 296/595] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:17.382158 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f64f0f-2d45-46e1-90ae-9fd73af406d2 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:17.387011 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11379015-263e-4244-ae3d-5f3c06283388 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:17.417088 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:17.440988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1691f6e-eef4-4312-b7cd-d3a9ee3101ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:17.441591 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1691f6e-eef4-4312-b7cd-d3a9ee3101ed None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:17.441901 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1691f6e-eef4-4312-b7cd-d3a9ee3101ed None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:17.442174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1691f6e-eef4-4312-b7cd-d3a9ee3101ed None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:17.443352 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1691f6e-eef4-4312-b7cd-d3a9ee3101ed None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:17.443653 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1691f6e-eef4-4312-b7cd-d3a9ee3101ed None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:17.445002 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1691f6e-eef4-4312-b7cd-d3a9ee3101ed None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:17.457529 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 300/596] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:17.458998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1691f6e-eef4-4312-b7cd-d3a9ee3101ed None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:17.481449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:17.521408 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f64f0f-2d45-46e1-90ae-9fd73af406d2 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:17.571550 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d54ae3-8a8f-402e-a82d-ee72a986ce50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:17.572925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d54ae3-8a8f-402e-a82d-ee72a986ce50 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:17.573456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d54ae3-8a8f-402e-a82d-ee72a986ce50 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:17.573924 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d54ae3-8a8f-402e-a82d-ee72a986ce50 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:17.575135 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1691f6e-eef4-4312-b7cd-d3a9ee3101ed None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:17.582417 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d54ae3-8a8f-402e-a82d-ee72a986ce50 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:17.583999 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d54ae3-8a8f-402e-a82d-ee72a986ce50 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:17.596290 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-50d54ae3-8a8f-402e-a82d-ee72a986ce50 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 27 04:07:17.596290 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:17.596290 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:17.596290 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:17.596290 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:17.596290 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:17.596290 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:17.596290 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:17.596290 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:17.596290 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:17.596290 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:17.596290 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:17.596290 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:17.596290 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:17.596290 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:17.596290 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:17.597498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:17.597498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:17.597498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:17.597498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:17.597498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:17.597498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:17.597498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:17.597498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:17.597498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:17.597498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:17.597498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:17.597498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:17.597498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:17.597498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:17.597498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:17.597498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:17.597498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:17.597498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:17.598657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:17.598657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:17.598657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:17.598657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:17.598657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:17.598657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:17.598657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:17.598657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:17.598657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 27 04:07:17.598657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:17.598657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 297/597] 10.222.0.26 () {56 vars in 1102 bytes} [Tue Sep 27 04:07:17 2022] GET /identity/v3/users/neutron => generated 85 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:17.599281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d54ae3-8a8f-402e-a82d-ee72a986ce50 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:17.623279 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb19807b-d877-489d-acf0-51a44842e010 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:17.624747 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb19807b-d877-489d-acf0-51a44842e010 None admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:17.625443 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb19807b-d877-489d-acf0-51a44842e010 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:17.625891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb19807b-d877-489d-acf0-51a44842e010 None admin] PATH_INFO: `/v3/projects/c830a3bfece6456486f4b094076b5364/users/8dcc53332e6b462baf349a72b19d2b3b/roles/20d83a8838004f60a32f8427577c7c35` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:17.647399 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb19807b-d877-489d-acf0-51a44842e010 None admin] RBAC: Authorizing `identity:create_grant(project_id=c830a3bfece6456486f4b094076b5364, user_id=8dcc53332e6b462baf349a72b19d2b3b, role_id=20d83a8838004f60a32f8427577c7c35)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:17.649405 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb19807b-d877-489d-acf0-51a44842e010 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:17.670652 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 301/598] 10.222.0.26 () {58 vars in 1439 bytes} [Tue Sep 27 04:07:17 2022] PUT /identity/v3/projects/c830a3bfece6456486f4b094076b5364/users/8dcc53332e6b462baf349a72b19d2b3b/roles/20d83a8838004f60a32f8427577c7c35 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:07:17.672313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb19807b-d877-489d-acf0-51a44842e010 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:17.691971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:17.740724 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d54ae3-8a8f-402e-a82d-ee72a986ce50 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:17.776566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:17.815231 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb19807b-d877-489d-acf0-51a44842e010 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:17.839094 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e4bbdf-2a2f-494a-8161-160a1279fe35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:17.839465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e4bbdf-2a2f-494a-8161-160a1279fe35 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:17.839750 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e4bbdf-2a2f-494a-8161-160a1279fe35 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:17.840028 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e4bbdf-2a2f-494a-8161-160a1279fe35 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:17.861715 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b34b10-664b-4794-b180-49879ac6aa98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:17.862823 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b34b10-664b-4794-b180-49879ac6aa98 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:17.863318 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b34b10-664b-4794-b180-49879ac6aa98 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:17.863658 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b34b10-664b-4794-b180-49879ac6aa98 None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:17.865086 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b34b10-664b-4794-b180-49879ac6aa98 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:17.865457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b34b10-664b-4794-b180-49879ac6aa98 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:17.866540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e4bbdf-2a2f-494a-8161-160a1279fe35 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:17.867082 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b34b10-664b-4794-b180-49879ac6aa98 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:17.867982 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e4bbdf-2a2f-494a-8161-160a1279fe35 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:17.885770 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-b4e4bbdf-2a2f-494a-8161-160a1279fe35 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:17.885770 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:17.885770 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:17.885770 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:17.885770 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:17.885770 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:17.885770 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:17.885770 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:17.885770 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:17.885770 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:17.885770 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:17.885770 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:17.885770 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:17.885770 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:17.885770 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:17.886910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:17.886910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:17.886910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:17.886910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:17.886910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:17.886910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:17.886910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:17.886910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:17.886910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:17.886910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:17.886910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:17.886910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:17.886910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:17.886910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:17.886910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:17.886910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:17.886910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:17.888149 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:17.888149 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:17.888149 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:17.888149 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:17.888149 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:17.888149 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:17.888149 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:17.888149 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:17.888149 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 298/599] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:17.891369 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 302/600] 10.222.0.26 () {56 vars in 1139 bytes} [Tue Sep 27 04:07:17 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:17.892820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e4bbdf-2a2f-494a-8161-160a1279fe35 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:17.896702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b34b10-664b-4794-b180-49879ac6aa98 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:18.018824 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e4bbdf-2a2f-494a-8161-160a1279fe35 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:18.027220 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b34b10-664b-4794-b180-49879ac6aa98 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:18.072033 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17aacf5b-fca4-43af-ae5b-32401a1e34d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:18.072642 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17aacf5b-fca4-43af-ae5b-32401a1e34d8 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:18.072918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17aacf5b-fca4-43af-ae5b-32401a1e34d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:18.073186 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17aacf5b-fca4-43af-ae5b-32401a1e34d8 None admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:18.073909 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17aacf5b-fca4-43af-ae5b-32401a1e34d8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:18.074213 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17aacf5b-fca4-43af-ae5b-32401a1e34d8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:18.075641 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17aacf5b-fca4-43af-ae5b-32401a1e34d8 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:18.089019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:18.099707 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 299/601] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:07:18.106061 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17aacf5b-fca4-43af-ae5b-32401a1e34d8 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:18.107599 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2fe204-b258-4b09-9697-c94ed75c3eb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:18.108238 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2fe204-b258-4b09-9697-c94ed75c3eb4 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:18.108516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2fe204-b258-4b09-9697-c94ed75c3eb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:18.108785 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2fe204-b258-4b09-9697-c94ed75c3eb4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:18.111992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:18.130105 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2fe204-b258-4b09-9697-c94ed75c3eb4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:18.131626 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2fe204-b258-4b09-9697-c94ed75c3eb4 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:18.149058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-ba2fe204-b258-4b09-9697-c94ed75c3eb4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:18.149058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:18.149058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:18.149058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:18.149058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:18.149058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:18.149058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:18.149058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:18.149058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:18.149058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:18.149058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:18.149058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:18.149058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:18.149058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:18.149058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:18.149058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:18.150102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:18.150102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:18.150102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:18.150102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:18.150102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:18.150102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:18.150102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:18.150102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:18.150102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:18.150102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:18.150102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:18.150102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:18.150102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:18.150102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:18.150102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:18.150102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:18.150102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:18.151423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:18.151423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:18.151423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:18.151423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 303/602] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:07:18.163173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2fe204-b258-4b09-9697-c94ed75c3eb4 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:18.232722 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17aacf5b-fca4-43af-ae5b-32401a1e34d8 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:18.268951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e4d04b-d4bc-4ffc-b552-170b3e78b54c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:18.269613 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e4d04b-d4bc-4ffc-b552-170b3e78b54c None admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:18.269922 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e4d04b-d4bc-4ffc-b552-170b3e78b54c None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:18.270197 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e4d04b-d4bc-4ffc-b552-170b3e78b54c None admin] PATH_INFO: `/v3/projects/c830a3bfece6456486f4b094076b5364/users/3970310fc7e34704b686e8d3bff3fdb2/roles/20d83a8838004f60a32f8427577c7c35` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:18.303673 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e4d04b-d4bc-4ffc-b552-170b3e78b54c None admin] RBAC: Authorizing `identity:create_grant(project_id=c830a3bfece6456486f4b094076b5364, user_id=3970310fc7e34704b686e8d3bff3fdb2, role_id=20d83a8838004f60a32f8427577c7c35)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:18.305254 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e4d04b-d4bc-4ffc-b552-170b3e78b54c None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:18.337135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 300/603] 10.222.0.26 () {58 vars in 1439 bytes} [Tue Sep 27 04:07:18 2022] PUT /identity/v3/projects/c830a3bfece6456486f4b094076b5364/users/3970310fc7e34704b686e8d3bff3fdb2/roles/20d83a8838004f60a32f8427577c7c35 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:07:18.338424 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf17ffdf-10e7-4660-a706-17c81a41f119 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:18.338740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf17ffdf-10e7-4660-a706-17c81a41f119 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:18.339010 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf17ffdf-10e7-4660-a706-17c81a41f119 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:18.340506 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 301/604] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:18.342443 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2fe204-b258-4b09-9697-c94ed75c3eb4 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:18.346105 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c5c012-8337-4c69-bad1-203077e2816b None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:18.346395 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c5c012-8337-4c69-bad1-203077e2816b None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:18.346670 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c5c012-8337-4c69-bad1-203077e2816b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:18.367042 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-b3c5c012-8337-4c69-bad1-203077e2816b None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:18.368315 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-b3c5c012-8337-4c69-bad1-203077e2816b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:18.424147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d48c86-0ea2-43a6-8829-20e3e0906f1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:18.424816 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d48c86-0ea2-43a6-8829-20e3e0906f1e None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:18.425045 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d48c86-0ea2-43a6-8829-20e3e0906f1e None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:18.425320 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d48c86-0ea2-43a6-8829-20e3e0906f1e None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:18.426178 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d48c86-0ea2-43a6-8829-20e3e0906f1e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:18.426481 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d48c86-0ea2-43a6-8829-20e3e0906f1e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:18.427034 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:18.428603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d48c86-0ea2-43a6-8829-20e3e0906f1e None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:18.461308 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 302/605] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:07:18.462017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 304/606] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:18.467992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c5c012-8337-4c69-bad1-203077e2816b None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:18.602723 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c5c012-8337-4c69-bad1-203077e2816b None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:18.638347 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41e94e5-5bca-4815-bc5b-2d8aaa6d1c3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:18.638909 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41e94e5-5bca-4815-bc5b-2d8aaa6d1c3a None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:18.639187 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41e94e5-5bca-4815-bc5b-2d8aaa6d1c3a None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:18.639460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41e94e5-5bca-4815-bc5b-2d8aaa6d1c3a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:18.648326 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b41e94e5-5bca-4815-bc5b-2d8aaa6d1c3a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:18.649818 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b41e94e5-5bca-4815-bc5b-2d8aaa6d1c3a None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:18.666491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-b41e94e5-5bca-4815-bc5b-2d8aaa6d1c3a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:18.666491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:18.666491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:18.666491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:18.666491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:18.666491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:18.666491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:18.666491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:18.666491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:18.666491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:18.666491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:18.666491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:18.666491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:18.666491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:18.666491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:18.667460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:18.667460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:18.667460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:18.667460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:18.667460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:18.667460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:18.667460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:18.667460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:18.667460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:18.667460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:18.667460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:18.667460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:18.667460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:18.667460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:18.667460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:18.667460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:18.667460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:18.668360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:18.668360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:18.668360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:18.668360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:18.668360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:18.668360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:18.668360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:18.668360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:18.668360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 303/607] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:18.673155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d48c86-0ea2-43a6-8829-20e3e0906f1e None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:18.732336 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:18.783849 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41e94e5-5bca-4815-bc5b-2d8aaa6d1c3a None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:18.818629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d48c86-0ea2-43a6-8829-20e3e0906f1e None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:18.873183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:18.875049 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d67f18-2d48-403f-9105-4663834a5bbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:18.875835 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d67f18-2d48-403f-9105-4663834a5bbb None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:18.876262 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d67f18-2d48-403f-9105-4663834a5bbb None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:18.877127 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d67f18-2d48-403f-9105-4663834a5bbb None admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:18.877364 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d67f18-2d48-403f-9105-4663834a5bbb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:18.877673 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d67f18-2d48-403f-9105-4663834a5bbb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:18.879491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d67f18-2d48-403f-9105-4663834a5bbb None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:18.893847 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 305/608] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:18.905684 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d67f18-2d48-403f-9105-4663834a5bbb None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:19.016518 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41e94e5-5bca-4815-bc5b-2d8aaa6d1c3a None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:19.050047 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:19.055361 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d67f18-2d48-403f-9105-4663834a5bbb None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:19.094590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65b7248-c35f-4b84-84a3-c027474800c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:19.095250 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65b7248-c35f-4b84-84a3-c027474800c3 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:19.095525 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65b7248-c35f-4b84-84a3-c027474800c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:19.096130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65b7248-c35f-4b84-84a3-c027474800c3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:19.101251 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67ec4e7-4f10-4149-924a-37975d19b923 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:19.102428 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67ec4e7-4f10-4149-924a-37975d19b923 None admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:19.103016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67ec4e7-4f10-4149-924a-37975d19b923 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:19.103776 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67ec4e7-4f10-4149-924a-37975d19b923 None admin] PATH_INFO: `/v3/projects/c830a3bfece6456486f4b094076b5364/users/cf29a8da103b411fbc79004eb371c05d/roles/20d83a8838004f60a32f8427577c7c35` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:19.122404 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67ec4e7-4f10-4149-924a-37975d19b923 None admin] RBAC: Authorizing `identity:create_grant(project_id=c830a3bfece6456486f4b094076b5364, user_id=cf29a8da103b411fbc79004eb371c05d, role_id=20d83a8838004f60a32f8427577c7c35)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:19.123005 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65b7248-c35f-4b84-84a3-c027474800c3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:19.124439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67ec4e7-4f10-4149-924a-37975d19b923 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:19.131842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-a65b7248-c35f-4b84-84a3-c027474800c3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:19.131842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:19.131842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:19.131842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:19.131842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:19.131842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:19.131842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:19.131842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:19.131842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:19.131842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:19.131842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:19.131842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:19.131842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:19.131842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:19.131842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:19.131842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:19.132790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:19.132790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:19.132790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:19.132790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:19.132790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:19.132790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:19.132790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:19.132790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:19.132790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:19.132790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:19.132790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:19.132790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:19.132790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:19.132790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:19.132790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:19.132790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:19.133618 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:19.133618 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:19.133618 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:19.133618 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:19.133618 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:19.133618 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:19.133618 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:19.133618 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:19.133618 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:19.133618 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:19.133618 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:19.133618 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:19.136276 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 304/609] 10.222.0.26 () {56 vars in 1102 bytes} [Tue Sep 27 04:07:18 2022] GET /identity/v3/roles/service => generated 85 bytes in 353 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:19.142365 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65b7248-c35f-4b84-84a3-c027474800c3 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:19.153095 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 306/610] 10.222.0.26 () {58 vars in 1439 bytes} [Tue Sep 27 04:07:18 2022] PUT /identity/v3/projects/c830a3bfece6456486f4b094076b5364/users/cf29a8da103b411fbc79004eb371c05d/roles/20d83a8838004f60a32f8427577c7c35 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:07:19.213442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07fed58-04b9-48df-81df-66fb5c9c163e None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:19.213818 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07fed58-04b9-48df-81df-66fb5c9c163e None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:19.214021 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07fed58-04b9-48df-81df-66fb5c9c163e None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:19.215478 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 307/611] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:07:19.221041 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a15786-2306-4d42-9bd1-b05fb9322153 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:19.221376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a15786-2306-4d42-9bd1-b05fb9322153 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:19.221584 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a15786-2306-4d42-9bd1-b05fb9322153 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:19.266817 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-46a15786-2306-4d42-9bd1-b05fb9322153 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:19.269655 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-46a15786-2306-4d42-9bd1-b05fb9322153 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:19.294028 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65b7248-c35f-4b84-84a3-c027474800c3 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:19.298447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:19.344297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd566be4-c389-4d20-9b4e-53560ff5bc2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:19.345787 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd566be4-c389-4d20-9b4e-53560ff5bc2e None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:19.346823 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd566be4-c389-4d20-9b4e-53560ff5bc2e None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:19.347302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd566be4-c389-4d20-9b4e-53560ff5bc2e None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:19.348199 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd566be4-c389-4d20-9b4e-53560ff5bc2e None admin] RBAC: Adding query filter params (name=service) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:19.348784 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd566be4-c389-4d20-9b4e-53560ff5bc2e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:19.350415 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd566be4-c389-4d20-9b4e-53560ff5bc2e None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:19.364852 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 305/612] 10.222.0.26 () {56 vars in 1103 bytes} [Tue Sep 27 04:07:19 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:07:19.370876 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd566be4-c389-4d20-9b4e-53560ff5bc2e None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:19.432321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 308/613] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Sep 27 04:07:19.484234 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd566be4-c389-4d20-9b4e-53560ff5bc2e None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:19.522419 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e743a0c-95d8-47b7-ba20-4f11a729571e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:19.523059 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e743a0c-95d8-47b7-ba20-4f11a729571e None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:19.523334 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e743a0c-95d8-47b7-ba20-4f11a729571e None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:19.523601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e743a0c-95d8-47b7-ba20-4f11a729571e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:19.537785 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e743a0c-95d8-47b7-ba20-4f11a729571e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:19.539766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e743a0c-95d8-47b7-ba20-4f11a729571e None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:19.560120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-0e743a0c-95d8-47b7-ba20-4f11a729571e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:19.560120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:19.560120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:19.560120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:19.560120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:19.560120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:19.560120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:19.560120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:19.560120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:19.560120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:19.560120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:19.560120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:19.560120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:19.560120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:19.560120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:19.560120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:19.561108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:19.561108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:19.561108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:19.561108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:19.561108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:19.561108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:19.561108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:19.561108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:19.561108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:19.561108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:19.561108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:19.561108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:19.561108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:19.561108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:19.561108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:19.561108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:19.561108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:19.562062 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:19.562062 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:19.562062 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:19.562062 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 306/614] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:07:19.566878 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a15786-2306-4d42-9bd1-b05fb9322153 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:19.695737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:19.718413 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a15786-2306-4d42-9bd1-b05fb9322153 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:19.775369 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5fa1de-2510-4263-b9c4-79172a47564a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:19.775975 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5fa1de-2510-4263-b9c4-79172a47564a None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:19.776251 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5fa1de-2510-4263-b9c4-79172a47564a None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:19.776513 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5fa1de-2510-4263-b9c4-79172a47564a None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:19.777149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac5fa1de-2510-4263-b9c4-79172a47564a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:19.777445 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac5fa1de-2510-4263-b9c4-79172a47564a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:19.778833 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac5fa1de-2510-4263-b9c4-79172a47564a None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:19.800962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 309/615] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:19.807027 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e743a0c-95d8-47b7-ba20-4f11a729571e None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:19.828946 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5fa1de-2510-4263-b9c4-79172a47564a None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:19.834444 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:19.982477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e743a0c-95d8-47b7-ba20-4f11a729571e None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:20.012101 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5fa1de-2510-4263-b9c4-79172a47564a None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:20.044811 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6765a6-3e76-4df8-918b-eadbf3dcb7f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:20.045438 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6765a6-3e76-4df8-918b-eadbf3dcb7f6 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:20.045709 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6765a6-3e76-4df8-918b-eadbf3dcb7f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:20.045998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6765a6-3e76-4df8-918b-eadbf3dcb7f6 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:20.059114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa6765a6-3e76-4df8-918b-eadbf3dcb7f6 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:20.060555 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa6765a6-3e76-4df8-918b-eadbf3dcb7f6 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:20.079512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-fa6765a6-3e76-4df8-918b-eadbf3dcb7f6 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 27 04:07:20.079512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:20.079512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:20.079512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:20.079512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:20.079512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:20.079512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:20.079512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:20.079512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:20.079512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:20.079512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:20.079512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:20.079512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:20.079512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:20.079512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:20.079512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:20.080506 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:20.080506 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:20.080506 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:20.080506 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:20.080506 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:20.080506 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:20.080506 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:20.080506 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:20.080506 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:20.080506 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:20.080506 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:20.080506 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:20.080506 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:20.080506 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:20.080506 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:20.080506 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:20.080506 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:20.080506 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:20.081456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:20.081456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:20.081456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:20.081456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:20.081456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:20.081456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:20.081456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:20.081456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:20.081456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 27 04:07:20.081456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:20.081456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 307/616] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:07:19 2022] GET /identity/v3/users/swift => generated 83 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:20.086830 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6765a6-3e76-4df8-918b-eadbf3dcb7f6 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:20.087078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f297a636-75aa-475d-885b-65f19d4741fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:20.087725 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f297a636-75aa-475d-885b-65f19d4741fd None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:20.088007 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f297a636-75aa-475d-885b-65f19d4741fd None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:20.088279 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f297a636-75aa-475d-885b-65f19d4741fd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:20.108279 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:20.117079 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f297a636-75aa-475d-885b-65f19d4741fd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:20.122634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-f297a636-75aa-475d-885b-65f19d4741fd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:20.122634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:20.122634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:20.122634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:20.122634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:20.122634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:20.122634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:20.122634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:20.122634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:20.122634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:20.122634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:20.122634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:20.122634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:20.122634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:20.122634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:20.122634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:20.123608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:20.123608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:20.123608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:20.123608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:20.123608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:20.123608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:20.123608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:20.123608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:20.123608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:20.123608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:20.123608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:20.123608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:20.123608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:20.123608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:20.123608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:20.123608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:20.124462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:20.124462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:20.124462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:20.124462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:20.124462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:20.124462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:20.124462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:20.124462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:20.124462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:20.124462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:20.124462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:20.124462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:20.124462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 310/617] 10.222.0.26 () {56 vars in 1102 bytes} [Tue Sep 27 04:07:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:20.134717 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f297a636-75aa-475d-885b-65f19d4741fd None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:20.222166 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:20.260998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6765a6-3e76-4df8-918b-eadbf3dcb7f6 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:20.280072 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f297a636-75aa-475d-885b-65f19d4741fd None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:20.299652 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0706be77-fd98-4568-b108-0f7d7bb682b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:20.300542 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0706be77-fd98-4568-b108-0f7d7bb682b5 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:20.300822 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0706be77-fd98-4568-b108-0f7d7bb682b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:20.301096 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0706be77-fd98-4568-b108-0f7d7bb682b5 None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:20.302161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0706be77-fd98-4568-b108-0f7d7bb682b5 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:20.302465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0706be77-fd98-4568-b108-0f7d7bb682b5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:20.303889 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0706be77-fd98-4568-b108-0f7d7bb682b5 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:20.325514 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 308/618] 10.222.0.26 () {56 vars in 1135 bytes} [Tue Sep 27 04:07:20 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:07:20.332487 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0706be77-fd98-4568-b108-0f7d7bb682b5 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:20.345600 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2fea35f-c58f-45c7-b66d-8611492afdac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:20.353767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fea35f-c58f-45c7-b66d-8611492afdac None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:20.353767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fea35f-c58f-45c7-b66d-8611492afdac None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:20.353767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fea35f-c58f-45c7-b66d-8611492afdac None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:20.353767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2fea35f-c58f-45c7-b66d-8611492afdac None admin] RBAC: Adding query filter params (name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:20.353767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2fea35f-c58f-45c7-b66d-8611492afdac None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:20.353767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2fea35f-c58f-45c7-b66d-8611492afdac None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:20.358114 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 311/619] 10.222.0.26 () {56 vars in 1103 bytes} [Tue Sep 27 04:07:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:20.359429 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be30a69f-0861-4ce8-aefa-726ce6479ca0 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:20.359701 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be30a69f-0861-4ce8-aefa-726ce6479ca0 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:20.359972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be30a69f-0861-4ce8-aefa-726ce6479ca0 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:20.361361 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 312/620] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:20.366964 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a5d4f5-26a3-41df-b2d6-41890e290f64 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:20.367314 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a5d4f5-26a3-41df-b2d6-41890e290f64 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:20.367651 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a5d4f5-26a3-41df-b2d6-41890e290f64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:20.387606 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-42a5d4f5-26a3-41df-b2d6-41890e290f64 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:20.388775 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-42a5d4f5-26a3-41df-b2d6-41890e290f64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:20.452439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:20.456210 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0706be77-fd98-4568-b108-0f7d7bb682b5 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:20.465138 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 313/621] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:07:20.466099 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a5d4f5-26a3-41df-b2d6-41890e290f64 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:20.524762 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d58bac-f608-410e-b35b-d92639b90d79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:20.525308 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d58bac-f608-410e-b35b-d92639b90d79 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:20.525583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d58bac-f608-410e-b35b-d92639b90d79 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:20.525973 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d58bac-f608-410e-b35b-d92639b90d79 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:20.531400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:20.549468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d58bac-f608-410e-b35b-d92639b90d79 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:20.551019 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d58bac-f608-410e-b35b-d92639b90d79 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:20.572955 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a5d4f5-26a3-41df-b2d6-41890e290f64 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:20.579044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-53d58bac-f608-410e-b35b-d92639b90d79 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:20.579044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:20.579044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:20.579044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:20.579044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:20.579044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:20.579044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:20.579044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:20.579044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:20.579044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:20.579044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:20.579044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:20.579044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:20.579044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:20.579044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:20.579044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:20.580019 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:20.580019 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:20.580019 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:20.580019 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:20.580019 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:20.580019 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:20.580019 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:20.580019 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:20.580019 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:20.580019 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:20.580019 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:20.580019 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:20.580019 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:20.580019 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:20.580019 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:20.580019 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:20.580019 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:20.580907 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:20.580907 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:20.580907 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:20.580907 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 309/622] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:07:20.586009 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d58bac-f608-410e-b35b-d92639b90d79 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:20.609771 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16502080-2707-4e3d-b91d-16fd6ec22d11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:20.610462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16502080-2707-4e3d-b91d-16fd6ec22d11 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:20.610739 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16502080-2707-4e3d-b91d-16fd6ec22d11 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:20.611003 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16502080-2707-4e3d-b91d-16fd6ec22d11 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:20.622341 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16502080-2707-4e3d-b91d-16fd6ec22d11 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:20.623886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16502080-2707-4e3d-b91d-16fd6ec22d11 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:20.640973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-16502080-2707-4e3d-b91d-16fd6ec22d11 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:20.640973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:20.640973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:20.640973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:20.640973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:20.640973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:20.640973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:20.640973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:20.640973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:20.640973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:20.640973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:20.640973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:20.640973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:20.640973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:20.640973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:20.640973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:20.641998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:20.641998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:20.641998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:20.641998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:20.641998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:20.641998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:20.641998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:20.641998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:20.641998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:20.641998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:20.641998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:20.641998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:20.641998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:20.641998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:20.641998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:20.641998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:20.641998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:20.642892 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:20.642892 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:20.642892 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:20.645876 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 314/623] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:20.647877 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16502080-2707-4e3d-b91d-16fd6ec22d11 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:20.697654 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:20.757510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16502080-2707-4e3d-b91d-16fd6ec22d11 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:20.792140 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e267eb-a986-4a1a-91ae-d02274ad2202 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:20.792742 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e267eb-a986-4a1a-91ae-d02274ad2202 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:20.793087 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e267eb-a986-4a1a-91ae-d02274ad2202 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:20.793363 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e267eb-a986-4a1a-91ae-d02274ad2202 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:20.794030 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e267eb-a986-4a1a-91ae-d02274ad2202 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:20.794329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e267eb-a986-4a1a-91ae-d02274ad2202 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:20.795785 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e267eb-a986-4a1a-91ae-d02274ad2202 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:20.812599 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 315/624] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:07:20.814674 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e267eb-a986-4a1a-91ae-d02274ad2202 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:20.832852 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d58bac-f608-410e-b35b-d92639b90d79 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:20.878233 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a553cce-da0f-4450-a312-6e3b60df74c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:20.879112 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a553cce-da0f-4450-a312-6e3b60df74c9 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:20.879384 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a553cce-da0f-4450-a312-6e3b60df74c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:20.879649 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a553cce-da0f-4450-a312-6e3b60df74c9 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:20.880879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a553cce-da0f-4450-a312-6e3b60df74c9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:20.881176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a553cce-da0f-4450-a312-6e3b60df74c9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:20.883411 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a553cce-da0f-4450-a312-6e3b60df74c9 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:20.897934 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 310/625] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:20.904164 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a553cce-da0f-4450-a312-6e3b60df74c9 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:20.970364 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:20.985434 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e267eb-a986-4a1a-91ae-d02274ad2202 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:20.987884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:21.024084 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1cfd14-8f2e-46f0-a398-d134b18dea79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:21.024948 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1cfd14-8f2e-46f0-a398-d134b18dea79 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:21.025367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1cfd14-8f2e-46f0-a398-d134b18dea79 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:21.025480 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a553cce-da0f-4450-a312-6e3b60df74c9 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:21.025943 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1cfd14-8f2e-46f0-a398-d134b18dea79 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:21.033901 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1cfd14-8f2e-46f0-a398-d134b18dea79 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:21.035633 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1cfd14-8f2e-46f0-a398-d134b18dea79 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:21.049067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-4e1cfd14-8f2e-46f0-a398-d134b18dea79 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 27 04:07:21.049067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:21.049067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:21.049067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:21.049067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:21.049067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:21.049067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:21.049067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:21.049067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:21.049067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:21.049067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:21.049067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:21.049067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:21.049067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:21.049067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:21.049067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:21.050226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:21.050226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:21.050226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:21.050226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:21.050226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:21.050226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:21.050226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:21.050226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:21.050226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:21.050226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:21.050226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:21.050226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:21.050226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:21.050226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:21.050226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:21.050226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:21.050226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:21.050226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:21.051328 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:21.051328 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:21.051328 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:21.051328 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:21.051328 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:21.051328 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:21.051328 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:21.051328 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:21.051328 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 27 04:07:21.051328 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:21.054436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 316/626] 10.222.0.26 () {56 vars in 1093 bytes} [Tue Sep 27 04:07:20 2022] GET /identity/v3/users/nova => generated 82 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:21.055367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1cfd14-8f2e-46f0-a398-d134b18dea79 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:21.066387 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a34daad-ebc4-406c-88e4-4bbf10b3fbb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:21.067019 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a34daad-ebc4-406c-88e4-4bbf10b3fbb4 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:21.067299 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a34daad-ebc4-406c-88e4-4bbf10b3fbb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:21.067600 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a34daad-ebc4-406c-88e4-4bbf10b3fbb4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:21.074293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a34daad-ebc4-406c-88e4-4bbf10b3fbb4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:21.075740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a34daad-ebc4-406c-88e4-4bbf10b3fbb4 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:21.086760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-0a34daad-ebc4-406c-88e4-4bbf10b3fbb4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:21.086760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:21.086760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:21.086760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:21.086760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:21.086760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:21.086760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:21.086760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:21.086760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:21.086760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:21.086760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:21.086760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:21.086760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:21.086760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:21.086760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:21.087856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:21.087856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:21.087856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:21.087856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:21.087856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:21.087856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:21.087856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:21.087856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:21.087856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:21.087856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:21.087856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:21.087856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:21.087856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:21.087856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:21.087856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:21.087856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:21.087856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:21.089200 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:21.089200 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:21.089200 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:21.089200 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:21.089200 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:21.089200 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:21.089200 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:21.089200 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:21.089200 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 311/627] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:07:21.089694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8928b1-0e1e-4bca-9c71-141fcd683b6a None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:21.089694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8928b1-0e1e-4bca-9c71-141fcd683b6a None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:21.089937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8928b1-0e1e-4bca-9c71-141fcd683b6a None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:21.091173 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 312/628] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:21 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:21.095520 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8928b1-0e1e-4bca-9c71-141fcd683b6a None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:21.167041 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1cfd14-8f2e-46f0-a398-d134b18dea79 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:21.201636 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b6995e-cdd1-480b-96ed-b042d7bb11a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:21.202298 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b6995e-cdd1-480b-96ed-b042d7bb11a4 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:21.202585 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b6995e-cdd1-480b-96ed-b042d7bb11a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:21.202852 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b6995e-cdd1-480b-96ed-b042d7bb11a4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:21.204112 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8928b1-0e1e-4bca-9c71-141fcd683b6a None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:21.212062 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:21.213436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b6995e-cdd1-480b-96ed-b042d7bb11a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:21.218879 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-15b6995e-cdd1-480b-96ed-b042d7bb11a4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:21.218879 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:21.218879 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:21.218879 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:21.218879 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:21.218879 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:21.218879 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:21.218879 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:21.218879 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:21.218879 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:21.218879 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:21.218879 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:21.218879 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:21.218879 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:21.218879 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:21.218879 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:21.219826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:21.219826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:21.219826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:21.219826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:21.219826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:21.219826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:21.219826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:21.219826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:21.219826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:21.219826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:21.219826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:21.219826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:21.219826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:21.219826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:21.219826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:21.219826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:21.220681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:21.220681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:21.220681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:21.220681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:21.220681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:21.220681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:21.220681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:21.220681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:21.220681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:21.220681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:21.220681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:21.220681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:21.220681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 317/629] 10.222.0.26 () {56 vars in 1102 bytes} [Tue Sep 27 04:07:21 2022] GET /identity/v3/roles/service => generated 85 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:21.222342 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b6995e-cdd1-480b-96ed-b042d7bb11a4 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:21.246991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d5a54a-b66f-4ef4-93d9-fb7334520a0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:21.247594 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d5a54a-b66f-4ef4-93d9-fb7334520a0f None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:21.247874 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d5a54a-b66f-4ef4-93d9-fb7334520a0f None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:21.248145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d5a54a-b66f-4ef4-93d9-fb7334520a0f None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:21.248865 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d5a54a-b66f-4ef4-93d9-fb7334520a0f None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:21.249322 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d5a54a-b66f-4ef4-93d9-fb7334520a0f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:21.250261 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:21.250619 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d5a54a-b66f-4ef4-93d9-fb7334520a0f None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:21.274298 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 313/630] 10.222.0.26 () {56 vars in 1133 bytes} [Tue Sep 27 04:07:21 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:21.274298 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588c7cd9-2167-418b-92bd-11f7cb79fd0a None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:21.274298 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588c7cd9-2167-418b-92bd-11f7cb79fd0a None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:21.274298 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588c7cd9-2167-418b-92bd-11f7cb79fd0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:21.292626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-588c7cd9-2167-418b-92bd-11f7cb79fd0a None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:21.293839 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-588c7cd9-2167-418b-92bd-11f7cb79fd0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:21.327519 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b6995e-cdd1-480b-96ed-b042d7bb11a4 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:21.369771 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 314/631] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:07:21.369771 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588c7cd9-2167-418b-92bd-11f7cb79fd0a None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:21.373047 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c18e44e-af7b-410f-aef7-963e1387a54d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:21.373768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c18e44e-af7b-410f-aef7-963e1387a54d None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:21.374134 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c18e44e-af7b-410f-aef7-963e1387a54d None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:21.374487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c18e44e-af7b-410f-aef7-963e1387a54d None admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:21.375257 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c18e44e-af7b-410f-aef7-963e1387a54d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:21.375634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c18e44e-af7b-410f-aef7-963e1387a54d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:21.377139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c18e44e-af7b-410f-aef7-963e1387a54d None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:21.390309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 318/632] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:21.391257 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c18e44e-af7b-410f-aef7-963e1387a54d None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:21.481770 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:21.498420 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:21.500155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c18e44e-af7b-410f-aef7-963e1387a54d None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:21.511577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588c7cd9-2167-418b-92bd-11f7cb79fd0a None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:21.545949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0135dc8b-5efb-49b5-a7a2-4736ef6c5c8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:21.546691 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0135dc8b-5efb-49b5-a7a2-4736ef6c5c8e None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:21.547048 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0135dc8b-5efb-49b5-a7a2-4736ef6c5c8e None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:21.547386 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0135dc8b-5efb-49b5-a7a2-4736ef6c5c8e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:21.558315 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111be7dd-cd2e-47d2-822b-9070e9b29da9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:21.559038 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111be7dd-cd2e-47d2-822b-9070e9b29da9 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:21.559387 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111be7dd-cd2e-47d2-822b-9070e9b29da9 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:21.560368 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111be7dd-cd2e-47d2-822b-9070e9b29da9 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:21.561075 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111be7dd-cd2e-47d2-822b-9070e9b29da9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:21.561379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111be7dd-cd2e-47d2-822b-9070e9b29da9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:21.563750 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111be7dd-cd2e-47d2-822b-9070e9b29da9 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:21.575547 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 315/633] 10.222.0.26 () {56 vars in 1103 bytes} [Tue Sep 27 04:07:21 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:21.576546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111be7dd-cd2e-47d2-822b-9070e9b29da9 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:21.634855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0135dc8b-5efb-49b5-a7a2-4736ef6c5c8e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:21.636274 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0135dc8b-5efb-49b5-a7a2-4736ef6c5c8e None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:21.684749 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111be7dd-cd2e-47d2-822b-9070e9b29da9 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:21.717984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29f5c40-3fec-41ee-8f96-70b9ac38cc17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:21.718625 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29f5c40-3fec-41ee-8f96-70b9ac38cc17 None admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:21.718899 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29f5c40-3fec-41ee-8f96-70b9ac38cc17 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:21.719169 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29f5c40-3fec-41ee-8f96-70b9ac38cc17 None admin] PATH_INFO: `/v3/projects/c830a3bfece6456486f4b094076b5364/users/181dd5dc720c4a97860852062b7b19c3/roles/20d83a8838004f60a32f8427577c7c35` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:21.725167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:21.732978 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29f5c40-3fec-41ee-8f96-70b9ac38cc17 None admin] RBAC: Authorizing `identity:create_grant(project_id=c830a3bfece6456486f4b094076b5364, user_id=181dd5dc720c4a97860852062b7b19c3, role_id=20d83a8838004f60a32f8427577c7c35)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:21.734431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29f5c40-3fec-41ee-8f96-70b9ac38cc17 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:21.742287 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 319/634] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:07:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:07:21.747454 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0135dc8b-5efb-49b5-a7a2-4736ef6c5c8e None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:21.749241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 316/635] 10.222.0.26 () {58 vars in 1439 bytes} [Tue Sep 27 04:07:21 2022] PUT /identity/v3/projects/c830a3bfece6456486f4b094076b5364/users/181dd5dc720c4a97860852062b7b19c3/roles/20d83a8838004f60a32f8427577c7c35 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:07:21.751405 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb65346-9002-4944-9937-e2baed39e9a2 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:21.751855 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb65346-9002-4944-9937-e2baed39e9a2 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:21.753203 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb65346-9002-4944-9937-e2baed39e9a2 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:21.760879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 317/636] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:21 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:21.760879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb65346-9002-4944-9937-e2baed39e9a2 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:21.775449 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:21.871071 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0135dc8b-5efb-49b5-a7a2-4736ef6c5c8e None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:21.890056 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb65346-9002-4944-9937-e2baed39e9a2 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:21.909929 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd3a079-fec0-4cd0-b897-b663d05a2949 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:21.910571 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd3a079-fec0-4cd0-b897-b663d05a2949 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:21.910846 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd3a079-fec0-4cd0-b897-b663d05a2949 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:21.911135 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd3a079-fec0-4cd0-b897-b663d05a2949 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:21.925815 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd3a079-fec0-4cd0-b897-b663d05a2949 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:21.928275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd3a079-fec0-4cd0-b897-b663d05a2949 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:21.934759 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f233f5a8-7df3-4005-81bc-3cb73fb288f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:21.935398 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f233f5a8-7df3-4005-81bc-3cb73fb288f8 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:21.935675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f233f5a8-7df3-4005-81bc-3cb73fb288f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:21.935945 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f233f5a8-7df3-4005-81bc-3cb73fb288f8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:21.945864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-ebd3a079-fec0-4cd0-b897-b663d05a2949 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:21.945864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:21.945864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:21.945864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:21.945864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:21.945864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:21.945864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:21.945864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:21.945864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:21.945864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:21.945864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:21.945864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:21.945864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:21.945864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:21.945864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:21.945864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:21.946851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:21.946851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:21.946851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:21.946851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:21.946851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:21.946851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:21.946851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:21.946851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:21.946851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:21.946851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:21.946851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:21.946851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:21.946851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:21.946851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:21.946851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:21.946851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:21.946851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:21.948167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:21.948167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:21.948167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:21.948167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 320/637] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:21.948167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd3a079-fec0-4cd0-b897-b663d05a2949 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:21.948457 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f233f5a8-7df3-4005-81bc-3cb73fb288f8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:21.957097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-f233f5a8-7df3-4005-81bc-3cb73fb288f8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:21.957097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:21.957097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:21.957097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:21.957097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:21.957097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:21.957097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:21.957097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:21.957097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:21.957097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:21.957097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:21.957097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:21.957097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:21.957097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:21.957097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:21.957097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:21.958712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:21.958712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:21.958712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:21.958712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:21.958712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:21.958712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:21.958712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:21.958712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:21.958712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:21.958712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:21.958712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:21.958712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:21.958712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:21.958712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:21.958712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:21.958712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:21.959579 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:21.959579 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:21.959579 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:21.959579 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:21.959579 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:21.959579 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:21.959579 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:21.959579 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:21.959579 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:21.959579 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:21.959579 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:21.959579 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:21.959579 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 318/638] 10.222.0.26 () {56 vars in 1102 bytes} [Tue Sep 27 04:07:21 2022] GET /identity/v3/roles/service => generated 85 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:21.960291 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd47e32-a1a3-4827-bc58-60cac7865a2d None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:21.960891 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd47e32-a1a3-4827-bc58-60cac7865a2d None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:21.960891 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd47e32-a1a3-4827-bc58-60cac7865a2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:21.979065 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-0cd47e32-a1a3-4827-bc58-60cac7865a2d None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:21.980233 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-0cd47e32-a1a3-4827-bc58-60cac7865a2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:21.995827 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:22.043543 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:22.053009 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 319/639] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:07:22.053870 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd47e32-a1a3-4827-bc58-60cac7865a2d None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:22.082051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd3a079-fec0-4cd0-b897-b663d05a2949 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:22.123090 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286ae9f1-7205-4bb1-94db-d84307b21d89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:22.123955 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286ae9f1-7205-4bb1-94db-d84307b21d89 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:22.124553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286ae9f1-7205-4bb1-94db-d84307b21d89 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:22.124956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286ae9f1-7205-4bb1-94db-d84307b21d89 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:22.138230 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286ae9f1-7205-4bb1-94db-d84307b21d89 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:22.140642 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286ae9f1-7205-4bb1-94db-d84307b21d89 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:22.157819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-286ae9f1-7205-4bb1-94db-d84307b21d89 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:22.157819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:22.157819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:22.157819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:22.157819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:22.157819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:22.157819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:22.157819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:22.157819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:22.157819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:22.157819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:22.157819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:22.157819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:22.157819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:22.157819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:22.157819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:22.159014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:22.159014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:22.159014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:22.159014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:22.159014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:22.159014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:22.159014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:22.159014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:22.159014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:22.159014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:22.159014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:22.159014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:22.159014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:22.159014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:22.159014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:22.159014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:22.159014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:22.160106 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:22.160106 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:22.160106 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:22.162164 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 321/640] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:22.163360 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286ae9f1-7205-4bb1-94db-d84307b21d89 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:22.175762 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd47e32-a1a3-4827-bc58-60cac7865a2d None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:22.218617 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256811dc-4913-46b5-9bd8-da32e1949c47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:22.219508 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256811dc-4913-46b5-9bd8-da32e1949c47 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:22.219947 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256811dc-4913-46b5-9bd8-da32e1949c47 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:22.220385 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256811dc-4913-46b5-9bd8-da32e1949c47 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:22.221159 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256811dc-4913-46b5-9bd8-da32e1949c47 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:22.221612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256811dc-4913-46b5-9bd8-da32e1949c47 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:22.223184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256811dc-4913-46b5-9bd8-da32e1949c47 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:22.235939 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 320/641] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:22.237096 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256811dc-4913-46b5-9bd8-da32e1949c47 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:22.274198 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:22.290677 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286ae9f1-7205-4bb1-94db-d84307b21d89 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:22.316778 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:22.340466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d728b39-b011-46fb-a884-67e5953c1918 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:22.341075 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d728b39-b011-46fb-a884-67e5953c1918 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:22.341861 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d728b39-b011-46fb-a884-67e5953c1918 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:22.342154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d728b39-b011-46fb-a884-67e5953c1918 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:22.342823 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d728b39-b011-46fb-a884-67e5953c1918 None admin] RBAC: Adding query filter params (name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:22.343126 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d728b39-b011-46fb-a884-67e5953c1918 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:22.345160 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d728b39-b011-46fb-a884-67e5953c1918 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:22.353779 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 322/642] 10.222.0.26 () {56 vars in 1103 bytes} [Tue Sep 27 04:07:22 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:22.353918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d728b39-b011-46fb-a884-67e5953c1918 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:22.373666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256811dc-4913-46b5-9bd8-da32e1949c47 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:22.416321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f299addf-c2e3-4f5e-87e1-76f77b64d4e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:22.416910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f299addf-c2e3-4f5e-87e1-76f77b64d4e0 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:22.417185 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f299addf-c2e3-4f5e-87e1-76f77b64d4e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:22.417451 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f299addf-c2e3-4f5e-87e1-76f77b64d4e0 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:22.418146 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f299addf-c2e3-4f5e-87e1-76f77b64d4e0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:22.418453 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f299addf-c2e3-4f5e-87e1-76f77b64d4e0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:22.419823 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f299addf-c2e3-4f5e-87e1-76f77b64d4e0 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:22.436440 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 321/643] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:22.436440 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f299addf-c2e3-4f5e-87e1-76f77b64d4e0 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:22.474450 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d728b39-b011-46fb-a884-67e5953c1918 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:22.513180 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a74d075-a36d-4c02-93b7-feb040846d27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:22.514747 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a74d075-a36d-4c02-93b7-feb040846d27 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:22.515236 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a74d075-a36d-4c02-93b7-feb040846d27 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:22.515723 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a74d075-a36d-4c02-93b7-feb040846d27 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:22.523517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a74d075-a36d-4c02-93b7-feb040846d27 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:22.525116 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a74d075-a36d-4c02-93b7-feb040846d27 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:22.532544 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f299addf-c2e3-4f5e-87e1-76f77b64d4e0 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:22.537491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-7a74d075-a36d-4c02-93b7-feb040846d27 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 27 04:07:22.537491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:22.537491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:22.537491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:22.537491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:22.537491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:22.537491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:22.537491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:22.537491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:22.537491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:22.537491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:22.537491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:22.537491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:22.537491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:22.537491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:22.537491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:22.538563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:22.538563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:22.538563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:22.538563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:22.538563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:22.538563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:22.538563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:22.538563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:22.538563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:22.538563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:22.538563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:22.538563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:22.538563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:22.538563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:22.538563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:22.538563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:22.538563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:22.538563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:22.539559 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:22.539559 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:22.539559 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:22.539559 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:22.539559 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:22.539559 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:22.539559 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:22.539559 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:22.539559 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 27 04:07:22.539559 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:22.539559 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 323/644] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:07:22 2022] GET /identity/v3/users/glance => generated 84 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:22.540186 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a74d075-a36d-4c02-93b7-feb040846d27 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:22.554319 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:22.555844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:22.574805 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc33940-4735-4859-bb23-e33fabc6a703 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:22.575544 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc33940-4735-4859-bb23-e33fabc6a703 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:22.575854 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc33940-4735-4859-bb23-e33fabc6a703 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:22.576124 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc33940-4735-4859-bb23-e33fabc6a703 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:22.588779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc33940-4735-4859-bb23-e33fabc6a703 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:22.594906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-4fc33940-4735-4859-bb23-e33fabc6a703 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:22.594906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:22.594906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:22.594906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:22.594906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:22.594906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:22.594906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:22.594906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:22.594906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:22.594906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:22.594906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:22.594906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:22.594906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:22.594906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:22.594906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:22.594906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:22.595859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:22.595859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:22.595859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:22.595859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:22.595859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:22.595859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:22.595859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:22.595859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:22.595859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:22.595859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:22.595859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:22.595859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:22.595859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:22.595859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:22.595859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:22.595859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:22.596846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:22.596846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:22.596846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:22.596846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:22.596846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:22.596846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:22.596846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:22.596846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:22.596846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:22.596846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:22.596846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:22.596846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:22.596846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 322/645] 10.222.0.26 () {56 vars in 1102 bytes} [Tue Sep 27 04:07:22 2022] GET /identity/v3/roles/service => generated 85 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:22.596846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc33940-4735-4859-bb23-e33fabc6a703 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:22.658365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a74d075-a36d-4c02-93b7-feb040846d27 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:22.694233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9660343-818c-4f16-9f9e-97817dc388dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:22.694856 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9660343-818c-4f16-9f9e-97817dc388dd None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:22.695525 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9660343-818c-4f16-9f9e-97817dc388dd None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:22.695797 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9660343-818c-4f16-9f9e-97817dc388dd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:22.706892 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9660343-818c-4f16-9f9e-97817dc388dd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:22.708337 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9660343-818c-4f16-9f9e-97817dc388dd None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:22.722698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc33940-4735-4859-bb23-e33fabc6a703 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:22.724155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-d9660343-818c-4f16-9f9e-97817dc388dd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:22.724155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:22.724155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:22.724155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:22.724155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:22.724155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:22.724155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:22.724155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:22.724155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:22.724155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:22.724155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:22.724155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:22.724155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:22.724155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:22.724155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:22.724155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:22.725143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:22.725143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:22.725143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:22.725143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:22.725143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:22.725143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:22.725143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:22.725143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:22.725143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:22.725143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:22.725143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:22.725143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:22.725143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:22.725143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:22.725143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:22.725143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:22.725143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:22.726076 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:22.726076 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:22.726076 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:22.726076 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 324/646] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:22.726305 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9660343-818c-4f16-9f9e-97817dc388dd None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:22.764743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade45fb7-2d25-4558-ac69-70908424440e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:22.765402 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade45fb7-2d25-4558-ac69-70908424440e None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:22.765683 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade45fb7-2d25-4558-ac69-70908424440e None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:22.765983 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade45fb7-2d25-4558-ac69-70908424440e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:22.772705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade45fb7-2d25-4558-ac69-70908424440e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:22.774178 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade45fb7-2d25-4558-ac69-70908424440e None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:22.786744 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-ade45fb7-2d25-4558-ac69-70908424440e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:22.786744 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:22.786744 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:22.786744 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:22.786744 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:22.786744 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:22.786744 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:22.786744 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:22.786744 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:22.786744 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:22.786744 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:22.786744 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:22.786744 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:22.786744 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:22.786744 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:22.787666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:22.787666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:22.787666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:22.787666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:22.787666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:22.787666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:22.787666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:22.787666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:22.787666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:22.787666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:22.787666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:22.787666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:22.787666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:22.787666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:22.787666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:22.787666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:22.787666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:22.788568 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:22.788568 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:22.788568 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:22.788568 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:22.788568 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:22.788568 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:22.788568 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:22.788568 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:22.788568 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 323/647] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:22.788568 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade45fb7-2d25-4558-ac69-70908424440e None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:22.795079 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:22.827355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:22.840133 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9660343-818c-4f16-9f9e-97817dc388dd None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:22.875953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cb2c54-ad33-4f93-aba9-b8537099f7b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:22.876547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cb2c54-ad33-4f93-aba9-b8537099f7b8 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:22.876866 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cb2c54-ad33-4f93-aba9-b8537099f7b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:22.877086 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cb2c54-ad33-4f93-aba9-b8537099f7b8 None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:22.877825 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cb2c54-ad33-4f93-aba9-b8537099f7b8 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:22.878128 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cb2c54-ad33-4f93-aba9-b8537099f7b8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:22.879803 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cb2c54-ad33-4f93-aba9-b8537099f7b8 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:22.892693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade45fb7-2d25-4558-ac69-70908424440e None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:22.902345 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 325/648] 10.222.0.26 () {56 vars in 1137 bytes} [Tue Sep 27 04:07:22 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:22.902345 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cb2c54-ad33-4f93-aba9-b8537099f7b8 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:22.930586 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db1d18a-a1d4-4d31-8fa1-33595c5159c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:22.931123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db1d18a-a1d4-4d31-8fa1-33595c5159c5 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:22.931403 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db1d18a-a1d4-4d31-8fa1-33595c5159c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:22.931677 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db1d18a-a1d4-4d31-8fa1-33595c5159c5 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:22.932328 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db1d18a-a1d4-4d31-8fa1-33595c5159c5 None admin] RBAC: Adding query filter params (name=service) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:22.932628 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db1d18a-a1d4-4d31-8fa1-33595c5159c5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:22.934015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db1d18a-a1d4-4d31-8fa1-33595c5159c5 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:22.941724 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 324/649] 10.222.0.26 () {56 vars in 1103 bytes} [Tue Sep 27 04:07:22 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:22.942574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db1d18a-a1d4-4d31-8fa1-33595c5159c5 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:23.008203 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cb2c54-ad33-4f93-aba9-b8537099f7b8 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:23.028599 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:23.044366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:23.045078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01928604-431b-49be-9e58-6262188498c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:23.045729 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01928604-431b-49be-9e58-6262188498c1 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:23.046051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01928604-431b-49be-9e58-6262188498c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:23.046389 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01928604-431b-49be-9e58-6262188498c1 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:23.047169 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01928604-431b-49be-9e58-6262188498c1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:23.047474 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01928604-431b-49be-9e58-6262188498c1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:23.048750 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db1d18a-a1d4-4d31-8fa1-33595c5159c5 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:23.048871 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01928604-431b-49be-9e58-6262188498c1 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:23.065014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 326/650] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:23.065014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01928604-431b-49be-9e58-6262188498c1 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:23.091086 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad13933-029d-4858-9979-cbb19dd61922 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:23.091688 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad13933-029d-4858-9979-cbb19dd61922 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:23.091959 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad13933-029d-4858-9979-cbb19dd61922 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:23.092226 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad13933-029d-4858-9979-cbb19dd61922 None admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:23.092896 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad13933-029d-4858-9979-cbb19dd61922 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:23.093198 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad13933-029d-4858-9979-cbb19dd61922 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:23.094629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad13933-029d-4858-9979-cbb19dd61922 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:23.106590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 325/651] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:23.107363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad13933-029d-4858-9979-cbb19dd61922 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:23.183117 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01928604-431b-49be-9e58-6262188498c1 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:23.207016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad13933-029d-4858-9979-cbb19dd61922 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:23.220316 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55412463-9be6-40f3-a6fb-ca8246e1c412 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:23.221205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55412463-9be6-40f3-a6fb-ca8246e1c412 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:23.221663 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55412463-9be6-40f3-a6fb-ca8246e1c412 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:23.222134 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55412463-9be6-40f3-a6fb-ca8246e1c412 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:23.240898 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e729c0a-5ce6-45d9-a66d-d8973054f647 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:23.241616 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e729c0a-5ce6-45d9-a66d-d8973054f647 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:23.241925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e729c0a-5ce6-45d9-a66d-d8973054f647 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:23.242199 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e729c0a-5ce6-45d9-a66d-d8973054f647 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:23.252132 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e729c0a-5ce6-45d9-a66d-d8973054f647 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:23.253856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e729c0a-5ce6-45d9-a66d-d8973054f647 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:23.268808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-0e729c0a-5ce6-45d9-a66d-d8973054f647 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:23.268808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:23.268808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:23.268808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:23.268808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:23.268808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:23.268808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:23.268808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:23.268808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:23.268808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:23.268808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:23.268808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:23.268808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:23.268808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:23.268808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:23.268808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:23.269798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:23.269798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:23.269798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:23.269798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:23.269798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:23.269798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:23.269798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:23.269798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:23.269798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:23.269798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:23.269798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:23.269798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:23.269798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:23.269798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:23.269798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:23.269798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:23.269798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:23.271295 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:23.271295 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:23.271295 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:23.271295 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 326/652] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:23.272218 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e729c0a-5ce6-45d9-a66d-d8973054f647 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:23.279586 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:23.285368 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:23.301930 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55412463-9be6-40f3-a6fb-ca8246e1c412 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:23.303357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55412463-9be6-40f3-a6fb-ca8246e1c412 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:23.375084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e729c0a-5ce6-45d9-a66d-d8973054f647 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:23.413942 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c95a88-c0d8-4878-a34c-533ec5aa3b0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:23.414639 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c95a88-c0d8-4878-a34c-533ec5aa3b0f None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:23.417443 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c95a88-c0d8-4878-a34c-533ec5aa3b0f None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:23.417443 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c95a88-c0d8-4878-a34c-533ec5aa3b0f None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:23.427105 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 327/653] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:07:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:07:23.427105 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55412463-9be6-40f3-a6fb-ca8246e1c412 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:23.429358 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c95a88-c0d8-4878-a34c-533ec5aa3b0f None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:23.430162 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c95a88-c0d8-4878-a34c-533ec5aa3b0f None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:23.441780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-66c95a88-c0d8-4878-a34c-533ec5aa3b0f None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 27 04:07:23.441780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:23.441780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:23.441780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:23.441780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:23.441780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:23.441780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:23.441780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:23.441780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:23.441780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:23.441780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:23.441780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:23.441780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:23.441780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:23.441780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:23.441780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:23.443307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:23.443307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:23.443307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:23.443307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:23.443307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:23.443307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:23.443307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:23.443307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:23.443307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:23.443307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:23.443307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:23.443307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:23.443307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:23.443307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:23.443307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:23.443307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:23.443307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:23.443307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:23.444260 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:23.444260 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:23.444260 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:23.444260 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:23.444260 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:23.444260 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:23.444260 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:23.444260 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:23.444260 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 27 04:07:23.444260 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:23.444260 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 327/654] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:07:23 2022] GET /identity/v3/users/cinder => generated 84 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:23.444260 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c95a88-c0d8-4878-a34c-533ec5aa3b0f None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:23.515990 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:23.518581 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:23.539795 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55412463-9be6-40f3-a6fb-ca8246e1c412 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:23.552737 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c95a88-c0d8-4878-a34c-533ec5aa3b0f None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:23.578495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75db5b9-deea-45f5-be4a-e78e1a297bcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:23.579088 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75db5b9-deea-45f5-be4a-e78e1a297bcb None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:23.579366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75db5b9-deea-45f5-be4a-e78e1a297bcb None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:23.579628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75db5b9-deea-45f5-be4a-e78e1a297bcb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:23.580343 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75db5b9-deea-45f5-be4a-e78e1a297bcb None admin] RBAC: Adding query filter params (role.id=20d83a8838004f60a32f8427577c7c35, scope.project.id=c830a3bfece6456486f4b094076b5364, user.id=a1586ee841cd44658e4d5feb25de7940) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:23.580647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75db5b9-deea-45f5-be4a-e78e1a297bcb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:23.582073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75db5b9-deea-45f5-be4a-e78e1a297bcb None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:23.587837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68fd166-09ea-42a6-a40e-15d6926b9990 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:23.588234 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 328/655] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:07:23 2022] GET /identity/v3/role_assignments?user.id=a1586ee841cd44658e4d5feb25de7940&scope.project.id=c830a3bfece6456486f4b094076b5364&role.id=20d83a8838004f60a32f8427577c7c35 => generated 608 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:23.588886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75db5b9-deea-45f5-be4a-e78e1a297bcb None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:23.598471 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68fd166-09ea-42a6-a40e-15d6926b9990 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:23.598803 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68fd166-09ea-42a6-a40e-15d6926b9990 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:23.609788 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68fd166-09ea-42a6-a40e-15d6926b9990 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:23.609788 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68fd166-09ea-42a6-a40e-15d6926b9990 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:23.609788 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68fd166-09ea-42a6-a40e-15d6926b9990 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:23.609788 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68fd166-09ea-42a6-a40e-15d6926b9990 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:23.622565 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 328/656] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:23.623467 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68fd166-09ea-42a6-a40e-15d6926b9990 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:23.720456 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75db5b9-deea-45f5-be4a-e78e1a297bcb None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:23.750614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68fd166-09ea-42a6-a40e-15d6926b9990 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:23.758858 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3aab4cb-d0aa-4615-ac21-b5dde2402961 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:23.759617 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3aab4cb-d0aa-4615-ac21-b5dde2402961 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:23.760042 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3aab4cb-d0aa-4615-ac21-b5dde2402961 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:23.760734 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3aab4cb-d0aa-4615-ac21-b5dde2402961 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:23.776209 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3aab4cb-d0aa-4615-ac21-b5dde2402961 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:23.778720 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3aab4cb-d0aa-4615-ac21-b5dde2402961 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:23.781764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:23.784799 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:23.790793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d62d9d2-309c-4ac3-b518-26b969ef236e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:23.792498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d62d9d2-309c-4ac3-b518-26b969ef236e None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:23.793003 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d62d9d2-309c-4ac3-b518-26b969ef236e None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:23.793489 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d62d9d2-309c-4ac3-b518-26b969ef236e None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:23.794565 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d62d9d2-309c-4ac3-b518-26b969ef236e None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:23.795095 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d62d9d2-309c-4ac3-b518-26b969ef236e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:23.796565 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d62d9d2-309c-4ac3-b518-26b969ef236e None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:23.796804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-e3aab4cb-d0aa-4615-ac21-b5dde2402961 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:23.796804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:23.796804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:23.796804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:23.796804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:23.796804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:23.796804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:23.796804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:23.796804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:23.796804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:23.796804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:23.796804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:23.796804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:23.796804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:23.796804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:23.796804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:23.797705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:23.797705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:23.797705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:23.797705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:23.797705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:23.797705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:23.797705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:23.797705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:23.797705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:23.797705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:23.797705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:23.797705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:23.797705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:23.797705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:23.797705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:23.797705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:23.797705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:23.798703 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:23.798703 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:23.798703 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:23.799323 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 329/657] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:23.802456 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3aab4cb-d0aa-4615-ac21-b5dde2402961 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:23.829566 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 329/658] 10.222.0.26 () {56 vars in 1137 bytes} [Tue Sep 27 04:07:23 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:23.830755 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d62d9d2-309c-4ac3-b518-26b969ef236e None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:23.927494 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3aab4cb-d0aa-4615-ac21-b5dde2402961 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:23.953009 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d62d9d2-309c-4ac3-b518-26b969ef236e None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:23.964919 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45dcfb6a-ce1d-4ff6-9939-73626ec1220c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:23.965603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45dcfb6a-ce1d-4ff6-9939-73626ec1220c None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:23.966207 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45dcfb6a-ce1d-4ff6-9939-73626ec1220c None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:23.966489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45dcfb6a-ce1d-4ff6-9939-73626ec1220c None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:23.976289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45dcfb6a-ce1d-4ff6-9939-73626ec1220c None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:23.978431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45dcfb6a-ce1d-4ff6-9939-73626ec1220c None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:23.987061 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fcfeb45-7edb-4f92-b5f1-1daf98021265 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:23.987661 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcfeb45-7edb-4f92-b5f1-1daf98021265 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:23.987938 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcfeb45-7edb-4f92-b5f1-1daf98021265 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:23.988205 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcfeb45-7edb-4f92-b5f1-1daf98021265 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:23.988845 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fcfeb45-7edb-4f92-b5f1-1daf98021265 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:23.989144 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fcfeb45-7edb-4f92-b5f1-1daf98021265 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:23.991089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fcfeb45-7edb-4f92-b5f1-1daf98021265 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:23.994218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-45dcfb6a-ce1d-4ff6-9939-73626ec1220c None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 27 04:07:23.994218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:23.994218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:23.994218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:23.994218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:23.994218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:23.994218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:23.994218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:23.994218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:23.994218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:23.994218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:23.994218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:23.994218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:23.994218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:23.994218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:23.994218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:23.995147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:23.995147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:23.995147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:23.995147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:23.995147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:23.995147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:23.995147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:23.995147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:23.995147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:23.995147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:23.995147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:23.995147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:23.995147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:23.995147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:23.995147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:23.995147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:23.995147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:23.995147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:24.000994 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:24.000994 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:24.000994 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:24.000994 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:24.000994 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:24.000994 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:24.000994 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:24.000994 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:24.000994 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 27 04:07:24.000994 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:24.000994 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 330/659] 10.222.0.26 () {56 vars in 1102 bytes} [Tue Sep 27 04:07:23 2022] GET /identity/v3/users/neutron => generated 85 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:24.000994 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45dcfb6a-ce1d-4ff6-9939-73626ec1220c None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:24.006700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 330/660] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:24.006700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9105f647-b913-4540-8597-666f0b00d037 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:24.006700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9105f647-b913-4540-8597-666f0b00d037 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:24.006700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9105f647-b913-4540-8597-666f0b00d037 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:24.006700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 331/661] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:24.007090 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9105f647-b913-4540-8597-666f0b00d037 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:24.062731 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:24.065468 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:24.112826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45dcfb6a-ce1d-4ff6-9939-73626ec1220c None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:24.117400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9105f647-b913-4540-8597-666f0b00d037 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:24.150296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be690f54-f5a1-44b8-8372-32a854582c59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:24.151061 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be690f54-f5a1-44b8-8372-32a854582c59 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:24.151503 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be690f54-f5a1-44b8-8372-32a854582c59 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:24.151821 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be690f54-f5a1-44b8-8372-32a854582c59 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:24.152903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856c67c3-00ea-435d-846d-a2a04910ce96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:24.153497 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856c67c3-00ea-435d-846d-a2a04910ce96 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:24.153797 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856c67c3-00ea-435d-846d-a2a04910ce96 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:24.154070 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856c67c3-00ea-435d-846d-a2a04910ce96 None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:24.154796 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856c67c3-00ea-435d-846d-a2a04910ce96 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:24.155096 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856c67c3-00ea-435d-846d-a2a04910ce96 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:24.156518 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856c67c3-00ea-435d-846d-a2a04910ce96 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:24.173889 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 332/662] 10.222.0.26 () {56 vars in 1139 bytes} [Tue Sep 27 04:07:24 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:24.174707 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856c67c3-00ea-435d-846d-a2a04910ce96 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:24.232744 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be690f54-f5a1-44b8-8372-32a854582c59 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:24.234216 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be690f54-f5a1-44b8-8372-32a854582c59 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:24.283915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856c67c3-00ea-435d-846d-a2a04910ce96 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:24.291859 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:24.293869 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:24.328459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd84182-4928-4ac9-a646-5858750f7a89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:24.329081 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd84182-4928-4ac9-a646-5858750f7a89 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:24.329354 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd84182-4928-4ac9-a646-5858750f7a89 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:24.329620 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd84182-4928-4ac9-a646-5858750f7a89 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:24.336102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd84182-4928-4ac9-a646-5858750f7a89 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:24.337167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 331/663] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:07:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:07:24.338267 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac50a2d3-45f5-4db5-89ee-09cb63457b17 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:24.338550 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac50a2d3-45f5-4db5-89ee-09cb63457b17 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:24.338823 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac50a2d3-45f5-4db5-89ee-09cb63457b17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:24.339031 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd84182-4928-4ac9-a646-5858750f7a89 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:24.351170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-bdd84182-4928-4ac9-a646-5858750f7a89 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:24.351170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:24.351170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:24.351170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:24.351170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:24.351170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:24.351170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:24.351170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:24.351170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:24.351170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:24.351170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:24.351170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:24.351170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:24.351170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:24.351170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:24.352104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:24.352104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:24.352104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:24.352104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:24.352104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:24.352104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:24.352104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:24.352104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:24.352104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:24.352104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:24.352104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:24.352104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:24.352104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:24.352104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:24.352104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:24.352104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:24.352104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:24.353016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:24.353016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:24.353016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:24.353016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:24.353016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:24.353016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:24.353016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:24.353016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:24.356910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 333/664] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:24.357586 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd84182-4928-4ac9-a646-5858750f7a89 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:24.357822 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-ac50a2d3-45f5-4db5-89ee-09cb63457b17 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:24.359013 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ac50a2d3-45f5-4db5-89ee-09cb63457b17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:24.437912 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 332/665] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:07:24.437912 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac50a2d3-45f5-4db5-89ee-09cb63457b17 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:24.474600 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd84182-4928-4ac9-a646-5858750f7a89 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:24.514248 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4a755f-1aee-4de8-aebe-224cc5debfec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:24.514765 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4a755f-1aee-4de8-aebe-224cc5debfec None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:24.515039 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4a755f-1aee-4de8-aebe-224cc5debfec None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:24.515306 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4a755f-1aee-4de8-aebe-224cc5debfec None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:24.536351 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:24.555500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac50a2d3-45f5-4db5-89ee-09cb63457b17 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:24.561952 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:24.596371 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f75af3-387e-45b4-8b67-fc0eca0f90eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:24.597364 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f75af3-387e-45b4-8b67-fc0eca0f90eb None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:24.597597 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f75af3-387e-45b4-8b67-fc0eca0f90eb None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:24.597998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f75af3-387e-45b4-8b67-fc0eca0f90eb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:24.604496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4a755f-1aee-4de8-aebe-224cc5debfec None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:24.606328 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4a755f-1aee-4de8-aebe-224cc5debfec None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:24.609133 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f75af3-387e-45b4-8b67-fc0eca0f90eb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:24.610966 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f75af3-387e-45b4-8b67-fc0eca0f90eb None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:24.629949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-f9f75af3-387e-45b4-8b67-fc0eca0f90eb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:24.629949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:24.629949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:24.629949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:24.629949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:24.629949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:24.629949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:24.629949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:24.629949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:24.629949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:24.629949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:24.629949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:24.629949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:24.629949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:24.629949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:24.629949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:24.631681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:24.631681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:24.631681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:24.631681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:24.631681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:24.631681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:24.631681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:24.631681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:24.631681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:24.631681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:24.631681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:24.631681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:24.631681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:24.631681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:24.631681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:24.631681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:24.631681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:24.632651 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:24.632651 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:24.632651 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:24.637687 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 333/666] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:24.638574 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f75af3-387e-45b4-8b67-fc0eca0f90eb None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:24.715354 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 334/667] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:07:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:07:24.716558 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4a755f-1aee-4de8-aebe-224cc5debfec None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:24.749823 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f75af3-387e-45b4-8b67-fc0eca0f90eb None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:24.784115 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:24.784261 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ed0c2a-2afc-4705-998f-f8369b299520 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:24.784675 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ed0c2a-2afc-4705-998f-f8369b299520 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:24.785197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ed0c2a-2afc-4705-998f-f8369b299520 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:24.785197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ed0c2a-2afc-4705-998f-f8369b299520 None admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:24.785999 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ed0c2a-2afc-4705-998f-f8369b299520 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:24.786898 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ed0c2a-2afc-4705-998f-f8369b299520 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:24.788425 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ed0c2a-2afc-4705-998f-f8369b299520 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:24.795495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:24.801817 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 334/668] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:24.803139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ed0c2a-2afc-4705-998f-f8369b299520 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:24.830984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4a755f-1aee-4de8-aebe-224cc5debfec None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:24.865861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ddd80c-1fb0-47be-bf02-fafc5c1c5519 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:24.866457 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ddd80c-1fb0-47be-bf02-fafc5c1c5519 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:24.866729 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ddd80c-1fb0-47be-bf02-fafc5c1c5519 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:24.866999 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ddd80c-1fb0-47be-bf02-fafc5c1c5519 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:24.867626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ddd80c-1fb0-47be-bf02-fafc5c1c5519 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:24.867920 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ddd80c-1fb0-47be-bf02-fafc5c1c5519 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:24.869256 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ddd80c-1fb0-47be-bf02-fafc5c1c5519 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:24.880559 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 335/669] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:24.881337 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ddd80c-1fb0-47be-bf02-fafc5c1c5519 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:24.918945 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ed0c2a-2afc-4705-998f-f8369b299520 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:24.956813 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c208471-d640-4986-869f-d99e5b22d2b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:24.957438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c208471-d640-4986-869f-d99e5b22d2b2 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:24.957715 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c208471-d640-4986-869f-d99e5b22d2b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:24.958037 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c208471-d640-4986-869f-d99e5b22d2b2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:24.969681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c208471-d640-4986-869f-d99e5b22d2b2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:24.971145 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c208471-d640-4986-869f-d99e5b22d2b2 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:24.985697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ddd80c-1fb0-47be-bf02-fafc5c1c5519 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:24.986296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-2c208471-d640-4986-869f-d99e5b22d2b2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:24.986296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:24.986296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:24.986296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:24.986296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:24.986296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:24.986296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:24.986296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:24.986296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:24.986296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:24.986296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:24.986296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:24.986296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:24.986296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:24.986296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:24.986296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:24.987210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:24.987210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:24.987210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:24.987210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:24.987210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:24.987210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:24.987210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:24.987210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:24.987210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:24.987210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:24.987210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:24.987210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:24.987210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:24.987210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:24.987210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:24.987210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:24.987210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:24.988121 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:24.988121 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:24.988121 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:24.988121 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 335/670] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:24.988736 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c208471-d640-4986-869f-d99e5b22d2b2 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:25.014762 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:25.024334 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de76d3c-0d21-4d31-8658-7635a174959e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:25.024964 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de76d3c-0d21-4d31-8658-7635a174959e None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:25.025238 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de76d3c-0d21-4d31-8658-7635a174959e None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:25.025512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de76d3c-0d21-4d31-8658-7635a174959e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:25.035831 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de76d3c-0d21-4d31-8658-7635a174959e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:25.041545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-1de76d3c-0d21-4d31-8658-7635a174959e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:25.041545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:25.041545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:25.041545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:25.041545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:25.041545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:25.041545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:25.041545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:25.041545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:25.041545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:25.041545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:25.041545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:25.041545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:25.041545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:25.041545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:25.041545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:25.042668 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:25.042668 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:25.042668 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:25.042668 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:25.042668 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:25.042668 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:25.042668 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:25.042668 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:25.042668 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:25.042668 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:25.042668 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:25.042668 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:25.042668 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:25.042668 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:25.042668 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:25.042668 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:25.043536 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:25.043536 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:25.043536 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:25.043536 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:25.043536 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:25.043536 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:25.043536 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:25.043536 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:25.043536 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:25.043536 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:25.043536 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:25.043536 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:25.043536 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 336/671] 10.222.0.26 () {56 vars in 1102 bytes} [Tue Sep 27 04:07:24 2022] GET /identity/v3/roles/service => generated 85 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:25.044259 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de76d3c-0d21-4d31-8658-7635a174959e None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:25.049368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:25.102012 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c208471-d640-4986-869f-d99e5b22d2b2 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:25.139665 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f642f00b-eed5-4fcb-a078-14ca91c54b7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:25.140263 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f642f00b-eed5-4fcb-a078-14ca91c54b7c None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:25.140537 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f642f00b-eed5-4fcb-a078-14ca91c54b7c None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:25.141216 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f642f00b-eed5-4fcb-a078-14ca91c54b7c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:25.142229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f642f00b-eed5-4fcb-a078-14ca91c54b7c None admin] RBAC: Adding query filter params (role.id=20d83a8838004f60a32f8427577c7c35, scope.project.id=c830a3bfece6456486f4b094076b5364, user.id=8dcc53332e6b462baf349a72b19d2b3b) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:25.142541 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f642f00b-eed5-4fcb-a078-14ca91c54b7c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:25.144105 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f642f00b-eed5-4fcb-a078-14ca91c54b7c None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:25.150526 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 336/672] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:07:24 2022] GET /identity/v3/role_assignments?user.id=8dcc53332e6b462baf349a72b19d2b3b&scope.project.id=c830a3bfece6456486f4b094076b5364&role.id=20d83a8838004f60a32f8427577c7c35 => generated 608 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:25.151382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f642f00b-eed5-4fcb-a078-14ca91c54b7c None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:25.153981 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de76d3c-0d21-4d31-8658-7635a174959e None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:25.194908 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d932f57-33c5-411f-97a6-3069dc911e5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:25.195694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d932f57-33c5-411f-97a6-3069dc911e5f None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:25.196110 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d932f57-33c5-411f-97a6-3069dc911e5f None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:25.196496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d932f57-33c5-411f-97a6-3069dc911e5f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:25.204003 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d932f57-33c5-411f-97a6-3069dc911e5f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:25.205588 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d932f57-33c5-411f-97a6-3069dc911e5f None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:25.218413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-1d932f57-33c5-411f-97a6-3069dc911e5f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:25.218413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:25.218413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:25.218413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:25.218413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:25.218413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:25.218413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:25.218413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:25.218413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:25.218413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:25.218413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:25.218413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:25.218413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:25.218413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:25.218413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:25.219425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:25.219425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:25.219425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:25.219425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:25.219425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:25.219425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:25.219425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:25.219425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:25.219425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:25.219425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:25.219425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:25.219425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:25.219425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:25.219425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:25.219425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:25.219425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:25.219425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:25.220303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:25.220303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:25.220303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:25.220303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:25.220303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:25.220303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:25.220303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:25.220303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:25.220303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 337/673] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:25.220784 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d932f57-33c5-411f-97a6-3069dc911e5f None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:25.274083 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:25.307315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f642f00b-eed5-4fcb-a078-14ca91c54b7c None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:25.328611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:25.336651 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d932f57-33c5-411f-97a6-3069dc911e5f None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:25.359669 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23007718-3eee-42cf-a595-59ca59ee8bd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:25.360470 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23007718-3eee-42cf-a595-59ca59ee8bd4 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:25.360866 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23007718-3eee-42cf-a595-59ca59ee8bd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:25.361203 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23007718-3eee-42cf-a595-59ca59ee8bd4 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:25.362128 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23007718-3eee-42cf-a595-59ca59ee8bd4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:25.362442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23007718-3eee-42cf-a595-59ca59ee8bd4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:25.364563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23007718-3eee-42cf-a595-59ca59ee8bd4 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:25.375282 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e331a4-29a2-4bbc-93ab-bb4336a94f41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:25.376080 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e331a4-29a2-4bbc-93ab-bb4336a94f41 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:25.376648 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e331a4-29a2-4bbc-93ab-bb4336a94f41 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:25.377133 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e331a4-29a2-4bbc-93ab-bb4336a94f41 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:25.378107 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e331a4-29a2-4bbc-93ab-bb4336a94f41 None admin] RBAC: Adding query filter params (name=service) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:25.378880 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e331a4-29a2-4bbc-93ab-bb4336a94f41 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:25.380469 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e331a4-29a2-4bbc-93ab-bb4336a94f41 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:25.381720 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 337/674] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:25.382594 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23007718-3eee-42cf-a595-59ca59ee8bd4 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:25.389559 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 338/675] 10.222.0.26 () {56 vars in 1103 bytes} [Tue Sep 27 04:07:25 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:25.390406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e331a4-29a2-4bbc-93ab-bb4336a94f41 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:25.507634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e331a4-29a2-4bbc-93ab-bb4336a94f41 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:25.512098 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23007718-3eee-42cf-a595-59ca59ee8bd4 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:25.517775 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:25.551225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7776797e-188f-4465-87c3-e70ac969910a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:25.552054 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7776797e-188f-4465-87c3-e70ac969910a None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:25.552243 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7776797e-188f-4465-87c3-e70ac969910a None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:25.552498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7776797e-188f-4465-87c3-e70ac969910a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:25.561727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7776797e-188f-4465-87c3-e70ac969910a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:25.563226 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7776797e-188f-4465-87c3-e70ac969910a None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:25.566761 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9529b433-87e1-4811-9a8d-eeba45302bf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:25.568414 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9529b433-87e1-4811-9a8d-eeba45302bf1 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:25.568842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9529b433-87e1-4811-9a8d-eeba45302bf1 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:25.569259 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9529b433-87e1-4811-9a8d-eeba45302bf1 None admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:25.570126 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9529b433-87e1-4811-9a8d-eeba45302bf1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:25.570596 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9529b433-87e1-4811-9a8d-eeba45302bf1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:25.572510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9529b433-87e1-4811-9a8d-eeba45302bf1 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:25.576279 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-7776797e-188f-4465-87c3-e70ac969910a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:25.576279 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:25.576279 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:25.576279 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:25.576279 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:25.576279 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:25.576279 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:25.576279 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:25.576279 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:25.576279 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:25.576279 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:25.576279 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:25.576279 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:25.576279 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:25.576279 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:25.577325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:25.577325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:25.577325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:25.577325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:25.577325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:25.577325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:25.577325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:25.577325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:25.577325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:25.577325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:25.577325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:25.577325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:25.577325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:25.577325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:25.577325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:25.577325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:25.577325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:25.578448 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:25.578448 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:25.578448 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:25.578448 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:25.578448 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:25.578448 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:25.578448 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:25.578448 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:25.584991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 339/676] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:25.586037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7776797e-188f-4465-87c3-e70ac969910a None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:25.594634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 338/677] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:25.595691 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9529b433-87e1-4811-9a8d-eeba45302bf1 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:25.641983 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:25.713738 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9529b433-87e1-4811-9a8d-eeba45302bf1 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:25.715585 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7776797e-188f-4465-87c3-e70ac969910a None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:25.748715 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70942490-4d79-43b4-858d-8711f93f3204 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:25.749338 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70942490-4d79-43b4-858d-8711f93f3204 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:25.749696 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70942490-4d79-43b4-858d-8711f93f3204 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:25.749951 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70942490-4d79-43b4-858d-8711f93f3204 None admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:25.750697 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70942490-4d79-43b4-858d-8711f93f3204 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:25.750998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70942490-4d79-43b4-858d-8711f93f3204 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:25.752395 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70942490-4d79-43b4-858d-8711f93f3204 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:25.763561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f8e482-0590-46d4-8188-98780b59b8ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:25.764735 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f8e482-0590-46d4-8188-98780b59b8ca None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:25.765204 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f8e482-0590-46d4-8188-98780b59b8ca None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:25.765653 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f8e482-0590-46d4-8188-98780b59b8ca None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:25.769859 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 339/678] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:25.771417 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70942490-4d79-43b4-858d-8711f93f3204 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:25.779731 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f8e482-0590-46d4-8188-98780b59b8ca None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:25.781973 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f8e482-0590-46d4-8188-98780b59b8ca None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:25.799779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-08f8e482-0590-46d4-8188-98780b59b8ca None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:25.799779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:25.799779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:25.799779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:25.799779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:25.799779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:25.799779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:25.799779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:25.799779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:25.799779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:25.799779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:25.799779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:25.799779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:25.799779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:25.799779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:25.799779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:25.801224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:25.801224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:25.801224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:25.801224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:25.801224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:25.801224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:25.801224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:25.801224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:25.801224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:25.801224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:25.801224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:25.801224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:25.801224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:25.801224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:25.801224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:25.801224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:25.801224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:25.802720 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:25.802720 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:25.802720 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:25.805140 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 340/679] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:25.807101 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f8e482-0590-46d4-8188-98780b59b8ca None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:25.833823 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:25.882458 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:25.889696 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70942490-4d79-43b4-858d-8711f93f3204 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:25.920379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f8e482-0590-46d4-8188-98780b59b8ca None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:25.926098 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec862fa-7361-4b2b-b64e-10dbd75f3eb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:25.927443 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec862fa-7361-4b2b-b64e-10dbd75f3eb0 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:25.927893 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec862fa-7361-4b2b-b64e-10dbd75f3eb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:25.928166 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec862fa-7361-4b2b-b64e-10dbd75f3eb0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:25.928891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec862fa-7361-4b2b-b64e-10dbd75f3eb0 None admin] RBAC: Adding query filter params (role.id=20d83a8838004f60a32f8427577c7c35, scope.project.id=c830a3bfece6456486f4b094076b5364, user.id=3970310fc7e34704b686e8d3bff3fdb2) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:25.929201 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec862fa-7361-4b2b-b64e-10dbd75f3eb0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:25.930718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec862fa-7361-4b2b-b64e-10dbd75f3eb0 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:25.937863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 340/680] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:07:25 2022] GET /identity/v3/role_assignments?user.id=3970310fc7e34704b686e8d3bff3fdb2&scope.project.id=c830a3bfece6456486f4b094076b5364&role.id=20d83a8838004f60a32f8427577c7c35 => generated 608 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:25.937863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec862fa-7361-4b2b-b64e-10dbd75f3eb0 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:25.971092 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ef034e-929d-4f34-b343-8060e1f41b0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:25.972386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ef034e-929d-4f34-b343-8060e1f41b0f None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:25.973129 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ef034e-929d-4f34-b343-8060e1f41b0f None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:25.973649 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ef034e-929d-4f34-b343-8060e1f41b0f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:25.975373 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ef034e-929d-4f34-b343-8060e1f41b0f None admin] RBAC: Adding query filter params (role.id=20d83a8838004f60a32f8427577c7c35, scope.project.id=c830a3bfece6456486f4b094076b5364, user.id=cf29a8da103b411fbc79004eb371c05d) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:25.975922 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ef034e-929d-4f34-b343-8060e1f41b0f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:25.978005 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ef034e-929d-4f34-b343-8060e1f41b0f None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:25.987238 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 341/681] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:07:25 2022] GET /identity/v3/role_assignments?user.id=cf29a8da103b411fbc79004eb371c05d&scope.project.id=c830a3bfece6456486f4b094076b5364&role.id=20d83a8838004f60a32f8427577c7c35 => generated 608 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:25.990214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7700003-b16a-43f6-9d12-276894236119 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:25.990852 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7700003-b16a-43f6-9d12-276894236119 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:25.991624 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7700003-b16a-43f6-9d12-276894236119 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:25.998937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 342/682] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:25 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:26.000319 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6192a469-d268-4c90-8724-32f2bcf662d2 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:26.000606 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6192a469-d268-4c90-8724-32f2bcf662d2 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:26.000880 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6192a469-d268-4c90-8724-32f2bcf662d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:26.022256 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-6192a469-d268-4c90-8724-32f2bcf662d2 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:26.023329 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6192a469-d268-4c90-8724-32f2bcf662d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:26.085710 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec862fa-7361-4b2b-b64e-10dbd75f3eb0 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:26.108443 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 343/683] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:07:26.149407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f24636-9e78-4f9d-95ac-ff4c7f96fd38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:26.150193 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f24636-9e78-4f9d-95ac-ff4c7f96fd38 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:26.150473 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f24636-9e78-4f9d-95ac-ff4c7f96fd38 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:26.150746 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f24636-9e78-4f9d-95ac-ff4c7f96fd38 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:26.151404 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f24636-9e78-4f9d-95ac-ff4c7f96fd38 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:26.151705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f24636-9e78-4f9d-95ac-ff4c7f96fd38 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:26.153150 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f24636-9e78-4f9d-95ac-ff4c7f96fd38 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:26.167407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 341/684] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:26.172668 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6192a469-d268-4c90-8724-32f2bcf662d2 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:26.192442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:26.291478 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6192a469-d268-4c90-8724-32f2bcf662d2 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:26.328050 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18aff3f-53b3-43f0-b12b-206f63551f6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:26.328949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18aff3f-53b3-43f0-b12b-206f63551f6c None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:26.329375 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18aff3f-53b3-43f0-b12b-206f63551f6c None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:26.329809 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18aff3f-53b3-43f0-b12b-206f63551f6c None admin] PATH_INFO: `/v3/users/swift` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:26.337280 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18aff3f-53b3-43f0-b12b-206f63551f6c None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:26.338906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18aff3f-53b3-43f0-b12b-206f63551f6c None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:26.352572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-d18aff3f-53b3-43f0-b12b-206f63551f6c None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 27 04:07:26.352572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:26.352572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:26.352572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:26.352572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:26.352572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:26.352572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:26.352572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:26.352572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:26.352572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:26.352572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:26.352572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:26.352572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:26.352572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:26.352572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:26.352572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:26.353708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:26.353708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:26.353708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:26.353708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:26.353708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:26.353708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:26.353708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:26.353708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:26.353708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:26.353708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:26.353708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:26.353708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:26.353708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:26.353708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:26.353708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:26.353708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:26.353708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:26.353708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:26.354896 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:26.354896 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:26.354896 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:26.354896 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:26.354896 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:26.354896 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:26.354896 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:26.354896 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:26.354896 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 27 04:07:26.354896 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:26.361361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 344/685] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:07:26 2022] GET /identity/v3/users/swift => generated 83 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:26.362944 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f24636-9e78-4f9d-95ac-ff4c7f96fd38 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:26.390321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:26.464584 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18aff3f-53b3-43f0-b12b-206f63551f6c None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:26.472431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f24636-9e78-4f9d-95ac-ff4c7f96fd38 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:26.525333 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc33c537-6fa0-4358-8aa1-aaf89e66abd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:26.526265 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc33c537-6fa0-4358-8aa1-aaf89e66abd8 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:26.526797 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc33c537-6fa0-4358-8aa1-aaf89e66abd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:26.527261 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc33c537-6fa0-4358-8aa1-aaf89e66abd8 None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:26.528270 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc33c537-6fa0-4358-8aa1-aaf89e66abd8 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:26.528746 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc33c537-6fa0-4358-8aa1-aaf89e66abd8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:26.530407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc33c537-6fa0-4358-8aa1-aaf89e66abd8 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:26.553844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 342/686] 10.222.0.26 () {56 vars in 1135 bytes} [Tue Sep 27 04:07:26 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:26.557647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc33c537-6fa0-4358-8aa1-aaf89e66abd8 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:26.629698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:26.646334 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:26.674847 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc33c537-6fa0-4358-8aa1-aaf89e66abd8 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:26.703722 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18aff3f-53b3-43f0-b12b-206f63551f6c None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:26.710128 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f790a20d-9f5d-41b7-98c0-abcf80c7aa6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:26.710814 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f790a20d-9f5d-41b7-98c0-abcf80c7aa6b None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:26.711217 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f790a20d-9f5d-41b7-98c0-abcf80c7aa6b None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:26.711562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f790a20d-9f5d-41b7-98c0-abcf80c7aa6b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:26.788151 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b7395e-a25e-4c99-9103-29a738b8f12a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:26.788870 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b7395e-a25e-4c99-9103-29a738b8f12a None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:26.790060 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b7395e-a25e-4c99-9103-29a738b8f12a None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:26.790772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b7395e-a25e-4c99-9103-29a738b8f12a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:26.802332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f790a20d-9f5d-41b7-98c0-abcf80c7aa6b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:26.804568 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f790a20d-9f5d-41b7-98c0-abcf80c7aa6b None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:26.816606 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b7395e-a25e-4c99-9103-29a738b8f12a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:26.823946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-d4b7395e-a25e-4c99-9103-29a738b8f12a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:07:26.823946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:26.823946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:26.823946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:26.823946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:26.823946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:26.823946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:26.823946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:26.823946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:26.823946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:26.823946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:26.823946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:26.823946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:26.823946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:26.823946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:26.823946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:26.825026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:26.825026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:26.825026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:26.825026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:26.825026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:26.825026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:26.825026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:26.825026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:26.825026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:26.825026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:26.825026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:26.825026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:26.825026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:26.825026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:26.825026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:26.825026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:26.826077 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:26.826077 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:26.826077 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:26.826077 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:26.826077 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:26.826077 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:26.826077 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:26.826077 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:26.826077 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:26.826077 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:26.826077 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:07:26.826077 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:26.829271 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 345/687] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:07:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 365 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:07:26.835355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b7395e-a25e-4c99-9103-29a738b8f12a None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:26.923174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:26.953056 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 343/688] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:07:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 27 04:07:26.954409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f790a20d-9f5d-41b7-98c0-abcf80c7aa6b None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:27.036739 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b7395e-a25e-4c99-9103-29a738b8f12a None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:27.059719 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f790a20d-9f5d-41b7-98c0-abcf80c7aa6b None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:27.097872 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa10d57-a3df-4fb4-96cc-22de7f3cf993 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:27.098855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa10d57-a3df-4fb4-96cc-22de7f3cf993 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:27.099955 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa10d57-a3df-4fb4-96cc-22de7f3cf993 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:27.100905 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa10d57-a3df-4fb4-96cc-22de7f3cf993 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:27.111670 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ae9265-5ce5-4362-9df3-c3c2cf902bd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:27.112416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ae9265-5ce5-4362-9df3-c3c2cf902bd5 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:27.112794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ae9265-5ce5-4362-9df3-c3c2cf902bd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:27.113149 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ae9265-5ce5-4362-9df3-c3c2cf902bd5 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:27.113956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ae9265-5ce5-4362-9df3-c3c2cf902bd5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:27.114384 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ae9265-5ce5-4362-9df3-c3c2cf902bd5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:27.115856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ae9265-5ce5-4362-9df3-c3c2cf902bd5 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:27.120689 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa10d57-a3df-4fb4-96cc-22de7f3cf993 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:27.121125 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:27.123311 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa10d57-a3df-4fb4-96cc-22de7f3cf993 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:27.138842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 346/689] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:07:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:27.144346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ae9265-5ce5-4362-9df3-c3c2cf902bd5 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:27.152852 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-2fa10d57-a3df-4fb4-96cc-22de7f3cf993 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:27.152852 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:27.152852 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:27.152852 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:27.152852 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:27.152852 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:27.152852 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:27.152852 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:27.152852 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:27.152852 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:27.152852 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:27.152852 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:27.152852 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:27.152852 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:27.152852 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:27.152852 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:27.154009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:27.154009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:27.154009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:27.154009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:27.154009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:27.154009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:27.154009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:27.154009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:27.154009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:27.154009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:27.154009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:27.154009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:27.154009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:27.154009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:27.154009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:27.154009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:27.154009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:27.155416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:27.155416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:27.155416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:27.167341 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 344/690] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:27.168895 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa10d57-a3df-4fb4-96cc-22de7f3cf993 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:27.228684 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:27.335767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa10d57-a3df-4fb4-96cc-22de7f3cf993 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:27.347525 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ae9265-5ce5-4362-9df3-c3c2cf902bd5 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:27.399702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e230a5-1274-4f33-82d8-e4013bd1ef13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:27.400308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e230a5-1274-4f33-82d8-e4013bd1ef13 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:27.400586 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e230a5-1274-4f33-82d8-e4013bd1ef13 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:27.400859 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e230a5-1274-4f33-82d8-e4013bd1ef13 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:27.401498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e230a5-1274-4f33-82d8-e4013bd1ef13 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:27.401825 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e230a5-1274-4f33-82d8-e4013bd1ef13 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:27.403204 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e230a5-1274-4f33-82d8-e4013bd1ef13 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:27.416791 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560e0558-253d-42e2-af5b-43311d386bda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:27.417428 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560e0558-253d-42e2-af5b-43311d386bda None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:27.417708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560e0558-253d-42e2-af5b-43311d386bda None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:27.418009 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560e0558-253d-42e2-af5b-43311d386bda None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:27.427383 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 345/691] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:07:27.434692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e230a5-1274-4f33-82d8-e4013bd1ef13 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:27.437976 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560e0558-253d-42e2-af5b-43311d386bda None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:27.439521 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560e0558-253d-42e2-af5b-43311d386bda None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:27.465436 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-560e0558-253d-42e2-af5b-43311d386bda None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:27.465436 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:27.465436 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:27.465436 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:27.465436 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:27.465436 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:27.465436 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:27.465436 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:27.465436 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:27.465436 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:27.465436 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:27.465436 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:27.465436 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:27.465436 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:27.465436 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:27.465436 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:27.466574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:27.466574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:27.466574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:27.466574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:27.466574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:27.466574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:27.466574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:27.466574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:27.466574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:27.466574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:27.466574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:27.466574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:27.466574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:27.466574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:27.466574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:27.466574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:27.466574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:27.467453 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:27.467453 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:27.467453 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:27.469040 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 347/692] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 323 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:27.473498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560e0558-253d-42e2-af5b-43311d386bda None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:27.534148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:27.548008 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e230a5-1274-4f33-82d8-e4013bd1ef13 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:27.570126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:27.620929 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cdeaa6-5d40-48a0-b41f-e3da2bcd5cb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:27.621565 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cdeaa6-5d40-48a0-b41f-e3da2bcd5cb3 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:27.621859 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cdeaa6-5d40-48a0-b41f-e3da2bcd5cb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:27.622127 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cdeaa6-5d40-48a0-b41f-e3da2bcd5cb3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:27.637213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cdeaa6-5d40-48a0-b41f-e3da2bcd5cb3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:27.638678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cdeaa6-5d40-48a0-b41f-e3da2bcd5cb3 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:27.658380 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560e0558-253d-42e2-af5b-43311d386bda None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:27.668963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-57cdeaa6-5d40-48a0-b41f-e3da2bcd5cb3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:27.668963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:27.668963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:27.668963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:27.668963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:27.668963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:27.668963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:27.668963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:27.668963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:27.668963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:27.668963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:27.668963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:27.668963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:27.668963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:27.668963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:27.670439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:27.670439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:27.670439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:27.670439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:27.670439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:27.670439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:27.670439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:27.670439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:27.670439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:27.670439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:27.670439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:27.670439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:27.670439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:27.670439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:27.670439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:27.670439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:27.670439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:27.671581 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:27.671581 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:27.671581 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:27.671581 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:27.671581 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:27.671581 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:27.671581 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:27.671581 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:27.675115 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 346/693] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:27.679049 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cdeaa6-5d40-48a0-b41f-e3da2bcd5cb3 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:27.711300 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e537ac0-66d0-4b39-8fa7-c32accb232a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:27.711905 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e537ac0-66d0-4b39-8fa7-c32accb232a7 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:27.712335 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e537ac0-66d0-4b39-8fa7-c32accb232a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:27.712612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e537ac0-66d0-4b39-8fa7-c32accb232a7 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:27.713716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e537ac0-66d0-4b39-8fa7-c32accb232a7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:27.714176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e537ac0-66d0-4b39-8fa7-c32accb232a7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:27.715547 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e537ac0-66d0-4b39-8fa7-c32accb232a7 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:27.737465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 348/694] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:07:27.740022 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1743f2ab-5fac-4eb9-9af0-c61c385c1c74 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:27.740300 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1743f2ab-5fac-4eb9-9af0-c61c385c1c74 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:27.740569 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1743f2ab-5fac-4eb9-9af0-c61c385c1c74 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:27.742075 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 349/695] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:27.742794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1743f2ab-5fac-4eb9-9af0-c61c385c1c74 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:27.810142 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cdeaa6-5d40-48a0-b41f-e3da2bcd5cb3 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:27.844230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1743f2ab-5fac-4eb9-9af0-c61c385c1c74 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:27.851423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58008345-6dfb-4846-9ff0-16adf7bbf91b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:27.852668 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58008345-6dfb-4846-9ff0-16adf7bbf91b None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:27.853182 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58008345-6dfb-4846-9ff0-16adf7bbf91b None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:27.853676 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58008345-6dfb-4846-9ff0-16adf7bbf91b None admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:27.854034 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:27.855165 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58008345-6dfb-4846-9ff0-16adf7bbf91b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:27.855678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58008345-6dfb-4846-9ff0-16adf7bbf91b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:27.857944 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58008345-6dfb-4846-9ff0-16adf7bbf91b None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:27.872798 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 347/696] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:27 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:07:27.874795 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a3f36f-6162-4bdb-be7e-1b169ffc3fef None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:27.875513 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a3f36f-6162-4bdb-be7e-1b169ffc3fef None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:27.875969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a3f36f-6162-4bdb-be7e-1b169ffc3fef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:27.883078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c065cfd5-8f7c-4b06-bafa-d20baea363af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:27.883744 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c065cfd5-8f7c-4b06-bafa-d20baea363af None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:27.884092 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c065cfd5-8f7c-4b06-bafa-d20baea363af None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:27.884413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c065cfd5-8f7c-4b06-bafa-d20baea363af None admin] PATH_INFO: `/v3/users/nova` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:27.892265 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c065cfd5-8f7c-4b06-bafa-d20baea363af None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:27.893770 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c065cfd5-8f7c-4b06-bafa-d20baea363af None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:27.897135 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-b7a3f36f-6162-4bdb-be7e-1b169ffc3fef None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:27.898640 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-b7a3f36f-6162-4bdb-be7e-1b169ffc3fef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:27.900740 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:27.906310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-c065cfd5-8f7c-4b06-bafa-d20baea363af None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 27 04:07:27.906310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:27.906310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:27.906310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:27.906310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:27.906310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:27.906310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:27.906310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:27.906310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:27.906310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:27.906310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:27.906310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:27.906310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:27.906310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:27.906310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:27.906310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:27.907257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:27.907257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:27.907257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:27.907257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:27.907257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:27.907257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:27.907257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:27.907257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:27.907257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:27.907257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:27.907257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:27.907257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:27.907257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:27.907257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:27.907257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:27.907257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:27.907257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:27.907257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:27.908222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:27.908222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:27.908222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:27.908222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:27.908222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:27.908222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:27.908222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:27.908222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:27.908222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 27 04:07:27.908222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:27.913160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 350/697] 10.222.0.26 () {56 vars in 1093 bytes} [Tue Sep 27 04:07:27 2022] GET /identity/v3/users/nova => generated 82 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:27.913780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c065cfd5-8f7c-4b06-bafa-d20baea363af None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:27.993872 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 348/698] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:07:27.995188 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a3f36f-6162-4bdb-be7e-1b169ffc3fef None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:28.041485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c065cfd5-8f7c-4b06-bafa-d20baea363af None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:28.117664 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94422f08-6f08-41e5-8fc3-d4c174bee62a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:28.118637 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94422f08-6f08-41e5-8fc3-d4c174bee62a None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:28.119136 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94422f08-6f08-41e5-8fc3-d4c174bee62a None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:28.119719 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94422f08-6f08-41e5-8fc3-d4c174bee62a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:28.120652 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94422f08-6f08-41e5-8fc3-d4c174bee62a None admin] RBAC: Adding query filter params (role.id=20d83a8838004f60a32f8427577c7c35, scope.project.id=c830a3bfece6456486f4b094076b5364, user.id=181dd5dc720c4a97860852062b7b19c3) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:28.121194 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94422f08-6f08-41e5-8fc3-d4c174bee62a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:28.123129 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94422f08-6f08-41e5-8fc3-d4c174bee62a None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:28.132582 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a3f36f-6162-4bdb-be7e-1b169ffc3fef None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:28.136065 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 351/699] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:07:27 2022] GET /identity/v3/role_assignments?user.id=181dd5dc720c4a97860852062b7b19c3&scope.project.id=c830a3bfece6456486f4b094076b5364&role.id=20d83a8838004f60a32f8427577c7c35 => generated 608 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:28.170804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc0a6c6-c87c-4910-bf97-c8db187baf67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:28.171412 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc0a6c6-c87c-4910-bf97-c8db187baf67 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:28.171688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc0a6c6-c87c-4910-bf97-c8db187baf67 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:28.171963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc0a6c6-c87c-4910-bf97-c8db187baf67 None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:28.172688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc0a6c6-c87c-4910-bf97-c8db187baf67 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:28.172989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc0a6c6-c87c-4910-bf97-c8db187baf67 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:28.174436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc0a6c6-c87c-4910-bf97-c8db187baf67 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:28.183761 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:28.201100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 349/700] 10.222.0.26 () {56 vars in 1133 bytes} [Tue Sep 27 04:07:27 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:28.202643 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94422f08-6f08-41e5-8fc3-d4c174bee62a None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:28.247632 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:28.338439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc0a6c6-c87c-4910-bf97-c8db187baf67 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:28.345065 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94422f08-6f08-41e5-8fc3-d4c174bee62a None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:28.394795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b70fbd-218e-4c7d-8da1-66c5bd4908ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:28.396354 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b70fbd-218e-4c7d-8da1-66c5bd4908ba None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:28.396839 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b70fbd-218e-4c7d-8da1-66c5bd4908ba None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:28.397362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b70fbd-218e-4c7d-8da1-66c5bd4908ba None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:28.498102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc0a6c6-c87c-4910-bf97-c8db187baf67 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:28.504782 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9b70fbd-218e-4c7d-8da1-66c5bd4908ba None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:28.506888 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9b70fbd-218e-4c7d-8da1-66c5bd4908ba None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:28.540503 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514ade86-3048-4fa3-baf2-b2f449169863 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:28.541153 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514ade86-3048-4fa3-baf2-b2f449169863 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:28.541425 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514ade86-3048-4fa3-baf2-b2f449169863 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:28.541691 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514ade86-3048-4fa3-baf2-b2f449169863 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:28.544819 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:28.554221 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514ade86-3048-4fa3-baf2-b2f449169863 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:28.555966 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514ade86-3048-4fa3-baf2-b2f449169863 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:28.573102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-514ade86-3048-4fa3-baf2-b2f449169863 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:28.573102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:28.573102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:28.573102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:28.573102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:28.573102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:28.573102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:28.573102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:28.573102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:28.573102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:28.573102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:28.573102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:28.573102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:28.573102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:28.573102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:28.573102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:28.574265 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:28.574265 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:28.574265 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:28.574265 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:28.574265 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:28.574265 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:28.574265 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:28.574265 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:28.574265 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:28.574265 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:28.574265 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:28.574265 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:28.574265 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:28.574265 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:28.574265 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:28.574265 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:28.574265 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:28.575455 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:28.575455 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:28.575455 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:28.577958 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 350/701] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:07:28.589169 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514ade86-3048-4fa3-baf2-b2f449169863 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:28.635622 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 352/702] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:07:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:07:28.640802 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b70fbd-218e-4c7d-8da1-66c5bd4908ba None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:28.651689 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:28.718531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514ade86-3048-4fa3-baf2-b2f449169863 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:28.751665 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b70fbd-218e-4c7d-8da1-66c5bd4908ba None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:28.757357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e674a8-1e99-4176-845f-0be2cbf290ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:28.757980 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e674a8-1e99-4176-845f-0be2cbf290ab None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:28.758261 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e674a8-1e99-4176-845f-0be2cbf290ab None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:28.758529 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e674a8-1e99-4176-845f-0be2cbf290ab None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:28.759167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e674a8-1e99-4176-845f-0be2cbf290ab None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:28.759463 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e674a8-1e99-4176-845f-0be2cbf290ab None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:28.761218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e674a8-1e99-4176-845f-0be2cbf290ab None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:28.775096 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 351/703] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:28.776128 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1a9fbb-7b7a-438c-a6cb-6b07faeb40af None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:28.776503 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1a9fbb-7b7a-438c-a6cb-6b07faeb40af None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:28.776778 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1a9fbb-7b7a-438c-a6cb-6b07faeb40af None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:28.781818 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 352/704] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:28.781818 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb0dd7c-2e1b-44b4-96fe-f1c7731f4e92 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:28.781818 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb0dd7c-2e1b-44b4-96fe-f1c7731f4e92 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:28.781818 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb0dd7c-2e1b-44b4-96fe-f1c7731f4e92 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:28.781818 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 353/705] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:28 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:28.781818 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb0dd7c-2e1b-44b4-96fe-f1c7731f4e92 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:28.791886 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:28.799167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0c2941-efce-4273-9559-0d2cfb0ef947 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:28.800319 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0c2941-efce-4273-9559-0d2cfb0ef947 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:28.800754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0c2941-efce-4273-9559-0d2cfb0ef947 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:28.801223 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0c2941-efce-4273-9559-0d2cfb0ef947 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:28.813305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0c2941-efce-4273-9559-0d2cfb0ef947 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:28.814753 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0c2941-efce-4273-9559-0d2cfb0ef947 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:28.831459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-9b0c2941-efce-4273-9559-0d2cfb0ef947 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:28.831459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:28.831459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:28.831459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:28.831459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:28.831459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:28.831459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:28.831459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:28.831459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:28.831459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:28.831459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:28.831459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:28.831459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:28.831459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:28.831459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:28.831459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:28.832431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:28.832431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:28.832431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:28.832431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:28.832431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:28.832431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:28.832431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:28.832431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:28.832431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:28.832431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:28.832431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:28.832431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:28.832431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:28.832431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:28.832431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:28.832431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:28.832431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:28.838031 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:28.838031 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:28.838031 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:28.838031 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 353/706] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:28.838031 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2445ec9-9aa2-47df-a6c3-6d69f6c14dde None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:28.838031 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2445ec9-9aa2-47df-a6c3-6d69f6c14dde None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:28.838031 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2445ec9-9aa2-47df-a6c3-6d69f6c14dde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:28.853817 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-d2445ec9-9aa2-47df-a6c3-6d69f6c14dde None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:28.854983 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d2445ec9-9aa2-47df-a6c3-6d69f6c14dde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:28.893886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb0dd7c-2e1b-44b4-96fe-f1c7731f4e92 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:28.904446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:28.925636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 354/707] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:07:28.927266 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903ff501-13e1-49fe-86dc-8284b565f04c None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:28.927550 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903ff501-13e1-49fe-86dc-8284b565f04c None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:28.927835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903ff501-13e1-49fe-86dc-8284b565f04c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:28.939569 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0fe476-a5ab-4b25-baff-5165a9f1a32b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:28.940461 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0fe476-a5ab-4b25-baff-5165a9f1a32b None admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:28.941085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0fe476-a5ab-4b25-baff-5165a9f1a32b None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:28.941602 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0fe476-a5ab-4b25-baff-5165a9f1a32b None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:28.943038 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0fe476-a5ab-4b25-baff-5165a9f1a32b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:28.944587 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0fe476-a5ab-4b25-baff-5165a9f1a32b None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:28.952564 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-903ff501-13e1-49fe-86dc-8284b565f04c None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:28.953997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-903ff501-13e1-49fe-86dc-8284b565f04c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:28.972403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 354/708] 10.222.0.26 () {60 vars in 1132 bytes} [Tue Sep 27 04:07:28 2022] POST /identity/v3/users => generated 254 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:07:28.973531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0fe476-a5ab-4b25-baff-5165a9f1a32b None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:29.103864 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 355/709] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:07:29.120745 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0fe476-a5ab-4b25-baff-5165a9f1a32b None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:29.165057 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a311c238-1f2b-4f87-be45-82e247340534 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:29.165661 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a311c238-1f2b-4f87-be45-82e247340534 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:29.165966 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a311c238-1f2b-4f87-be45-82e247340534 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:29.166236 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a311c238-1f2b-4f87-be45-82e247340534 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:29.166890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a311c238-1f2b-4f87-be45-82e247340534 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:29.167188 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a311c238-1f2b-4f87-be45-82e247340534 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:29.168573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a311c238-1f2b-4f87-be45-82e247340534 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:29.188339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 355/710] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:29.194667 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903ff501-13e1-49fe-86dc-8284b565f04c None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:29.219655 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a311c238-1f2b-4f87-be45-82e247340534 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:29.249566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:29.256911 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:29.316370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903ff501-13e1-49fe-86dc-8284b565f04c None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:29.338801 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a311c238-1f2b-4f87-be45-82e247340534 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:29.354706 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4e7558-823c-4877-8e38-211e335c13a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:29.355421 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4e7558-823c-4877-8e38-211e335c13a9 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:29.355835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4e7558-823c-4877-8e38-211e335c13a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:29.356168 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4e7558-823c-4877-8e38-211e335c13a9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:29.363338 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df4e7558-823c-4877-8e38-211e335c13a9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:29.365546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df4e7558-823c-4877-8e38-211e335c13a9 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:29.378406 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f19d67-3c19-457c-8aaf-cb1be2471a08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:29.379296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f19d67-3c19-457c-8aaf-cb1be2471a08 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:29.379572 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f19d67-3c19-457c-8aaf-cb1be2471a08 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:29.379675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-df4e7558-823c-4877-8e38-211e335c13a9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:29.379675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:29.379675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:29.379675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:29.379675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:29.379675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:29.379675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:29.379675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:29.379675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:29.379675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:29.379675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:29.379675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:29.379675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:29.379675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:29.379675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:29.380560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:29.380560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:29.380560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:29.380560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:29.380560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:29.380560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:29.380560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:29.380560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:29.380560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:29.380560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:29.380560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:29.380560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:29.380560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:29.380560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:29.380560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:29.380560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:29.380560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:29.381429 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:29.381429 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:29.381429 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:29.381429 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:29.381429 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:29.381429 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:29.381429 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:29.381429 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:29.381893 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f19d67-3c19-457c-8aaf-cb1be2471a08 None admin] PATH_INFO: `/v3/services/network` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:29.381893 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f19d67-3c19-457c-8aaf-cb1be2471a08 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:29.382029 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 356/711] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:29.382100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f19d67-3c19-457c-8aaf-cb1be2471a08 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:29.386625 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4e7558-823c-4877-8e38-211e335c13a9 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:29.394449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-52f19d67-3c19-457c-8aaf-cb1be2471a08 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 27 04:07:29.394449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:29.394449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:07:29.394449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:07:29.394449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:07:29.394449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:07:29.394449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:07:29.394449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:29.394449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 27 04:07:29.394449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:29.394449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:07:29.394449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:29.394449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:29.394449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:29.394449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:29.394449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:29.394449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:29.396245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:29.396245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:29.396245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:29.396245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:29.396245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:29.396245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:29.396245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:07:29.396245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:07:29.396245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:07:29.396245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:07:29.396245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:29.396245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:29.396245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:29.396245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:29.396245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:29.396245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:29.397996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:29.397996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:29.397996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:29.397996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:29.397996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:29.397996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:29.397996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:29.397996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:29.397996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:29.397996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:29.397996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:07:29.397996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:29.397996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 27 04:07:29.397996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:29.403973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 356/712] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:29 2022] GET /identity/v3/services/network => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:29.403973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f19d67-3c19-457c-8aaf-cb1be2471a08 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:29.508009 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4e7558-823c-4877-8e38-211e335c13a9 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:29.512929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:29.519575 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:29.538976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f19d67-3c19-457c-8aaf-cb1be2471a08 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:29.552839 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350b1f60-6f7e-44a9-824c-8978ce1293cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:29.553435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350b1f60-6f7e-44a9-824c-8978ce1293cb None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:29.553811 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350b1f60-6f7e-44a9-824c-8978ce1293cb None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:29.554162 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350b1f60-6f7e-44a9-824c-8978ce1293cb None admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:29.555134 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350b1f60-6f7e-44a9-824c-8978ce1293cb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:29.555260 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350b1f60-6f7e-44a9-824c-8978ce1293cb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:29.556735 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350b1f60-6f7e-44a9-824c-8978ce1293cb None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:29.572374 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 357/713] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:07:29.578784 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350b1f60-6f7e-44a9-824c-8978ce1293cb None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:29.584940 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baaab93c-bb79-422a-970c-5db5bfc6f6e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:29.585827 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baaab93c-bb79-422a-970c-5db5bfc6f6e6 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:29.586279 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baaab93c-bb79-422a-970c-5db5bfc6f6e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:29.586719 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baaab93c-bb79-422a-970c-5db5bfc6f6e6 None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:29.587598 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baaab93c-bb79-422a-970c-5db5bfc6f6e6 None admin] RBAC: Adding query filter params (name=network) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:29.588069 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baaab93c-bb79-422a-970c-5db5bfc6f6e6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:29.589645 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baaab93c-bb79-422a-970c-5db5bfc6f6e6 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:29.602047 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 357/714] 10.222.0.26 () {56 vars in 1112 bytes} [Tue Sep 27 04:07:29 2022] GET /identity/v3/services?name=network => generated 124 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:29.602047 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baaab93c-bb79-422a-970c-5db5bfc6f6e6 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:29.705380 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350b1f60-6f7e-44a9-824c-8978ce1293cb None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:29.725044 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baaab93c-bb79-422a-970c-5db5bfc6f6e6 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:29.742042 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f331364f-0c3d-4204-9bcd-f33eb2f491dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:29.742963 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f331364f-0c3d-4204-9bcd-f33eb2f491dd None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:29.743359 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f331364f-0c3d-4204-9bcd-f33eb2f491dd None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:29.743728 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f331364f-0c3d-4204-9bcd-f33eb2f491dd None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:29.744782 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f331364f-0c3d-4204-9bcd-f33eb2f491dd None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:29.746535 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f331364f-0c3d-4204-9bcd-f33eb2f491dd None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:29.761314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-f331364f-0c3d-4204-9bcd-f33eb2f491dd None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 27 04:07:29.761314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:29.761314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:07:29.761314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:07:29.761314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:07:29.761314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:07:29.761314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:07:29.761314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:29.761314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 27 04:07:29.761314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:29.761314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:07:29.761314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:29.761314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:29.761314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:29.761314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:29.761314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:29.761314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:29.762919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:29.762919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:29.762919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:29.762919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:29.762919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:29.762919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:29.762919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:07:29.762919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:07:29.762919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:07:29.762919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:07:29.762919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:29.762919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:29.762919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:29.762919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:29.762919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:29.762919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:29.763792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:29.763792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:29.763792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:29.763792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:29.763792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:29.763792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:29.763792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:29.763792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:29.763792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:29.763792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:29.763792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:07:29.763792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:29.763792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 27 04:07:29.763792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:29.765825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 358/715] 10.222.0.26 () {56 vars in 1129 bytes} [Tue Sep 27 04:07:29 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:29.766857 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f331364f-0c3d-4204-9bcd-f33eb2f491dd None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:29.774455 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf143a5-7d63-4d79-be03-e193f8c86749 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:29.775394 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf143a5-7d63-4d79-be03-e193f8c86749 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:29.775890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf143a5-7d63-4d79-be03-e193f8c86749 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:29.776438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf143a5-7d63-4d79-be03-e193f8c86749 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:29.777831 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf143a5-7d63-4d79-be03-e193f8c86749 None admin] RBAC: Adding query filter params (role.id=e6a31562a18649f0809202c36a8e18ef, scope.project.id=c830a3bfece6456486f4b094076b5364, user.id=a1586ee841cd44658e4d5feb25de7940) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:29.778340 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf143a5-7d63-4d79-be03-e193f8c86749 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:29.780388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf143a5-7d63-4d79-be03-e193f8c86749 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:29.788686 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 358/716] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:07:29 2022] GET /identity/v3/role_assignments?user.id=a1586ee841cd44658e4d5feb25de7940&scope.project.id=c830a3bfece6456486f4b094076b5364&role.id=e6a31562a18649f0809202c36a8e18ef => generated 259 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:29.789867 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf143a5-7d63-4d79-be03-e193f8c86749 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:29.814140 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:29.824153 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:29.940092 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf143a5-7d63-4d79-be03-e193f8c86749 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:29.976894 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075db1a1-f9de-4b47-8eaf-16e9c0d32dc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:29.977536 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075db1a1-f9de-4b47-8eaf-16e9c0d32dc5 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:29.977863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075db1a1-f9de-4b47-8eaf-16e9c0d32dc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:29.978136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075db1a1-f9de-4b47-8eaf-16e9c0d32dc5 None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:29.978784 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075db1a1-f9de-4b47-8eaf-16e9c0d32dc5 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:29.979159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075db1a1-f9de-4b47-8eaf-16e9c0d32dc5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:29.980438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075db1a1-f9de-4b47-8eaf-16e9c0d32dc5 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:29.985762 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 359/717] 10.222.0.26 () {56 vars in 1124 bytes} [Tue Sep 27 04:07:29 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:29.991405 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075db1a1-f9de-4b47-8eaf-16e9c0d32dc5 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:30.007416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f331364f-0c3d-4204-9bcd-f33eb2f491dd None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:30.060376 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd844b2-9347-46a0-9871-57f52d12529e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:30.060984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd844b2-9347-46a0-9871-57f52d12529e None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:30.061271 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd844b2-9347-46a0-9871-57f52d12529e None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:30.061646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd844b2-9347-46a0-9871-57f52d12529e None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:30.062468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd844b2-9347-46a0-9871-57f52d12529e None admin] RBAC: Adding query filter params (type=network) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:30.062774 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd844b2-9347-46a0-9871-57f52d12529e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:30.064384 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd844b2-9347-46a0-9871-57f52d12529e None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:30.072808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 359/718] 10.222.0.26 () {56 vars in 1112 bytes} [Tue Sep 27 04:07:29 2022] GET /identity/v3/services?type=network => generated 124 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:30.098305 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:30.117091 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075db1a1-f9de-4b47-8eaf-16e9c0d32dc5 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:30.154367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0689d7-c6f8-4320-82ec-57af259d9c31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:30.154976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0689d7-c6f8-4320-82ec-57af259d9c31 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:30.155255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0689d7-c6f8-4320-82ec-57af259d9c31 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:30.155530 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0689d7-c6f8-4320-82ec-57af259d9c31 None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:30.156168 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0689d7-c6f8-4320-82ec-57af259d9c31 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:30.156469 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0689d7-c6f8-4320-82ec-57af259d9c31 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:30.157947 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0689d7-c6f8-4320-82ec-57af259d9c31 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:30.164490 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 360/719] 10.222.0.26 () {56 vars in 1124 bytes} [Tue Sep 27 04:07:29 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:30.976623 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b0e48e-5d28-4c42-9154-7577637c8353 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:30.977143 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b0e48e-5d28-4c42-9154-7577637c8353 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:30.977143 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b0e48e-5d28-4c42-9154-7577637c8353 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:30.978772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 360/720] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:30.995678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9dd9fb-439c-43ac-bd3d-d24b59af7177 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:30.996615 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9dd9fb-439c-43ac-bd3d-d24b59af7177 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:30.997083 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9dd9fb-439c-43ac-bd3d-d24b59af7177 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:31.021733 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-0f9dd9fb-439c-43ac-bd3d-d24b59af7177 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:31.023313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0f9dd9fb-439c-43ac-bd3d-d24b59af7177 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:31.111374 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 361/721] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:07:31.576036 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b0e48e-5d28-4c42-9154-7577637c8353 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:31.706874 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:31.717340 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b0e48e-5d28-4c42-9154-7577637c8353 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:31.737734 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc4446c-4d5f-4f61-a03e-1197ce5e84ca None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:31.738409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc4446c-4d5f-4f61-a03e-1197ce5e84ca None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:31.738882 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc4446c-4d5f-4f61-a03e-1197ce5e84ca None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:31.744992 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 362/722] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:31.746777 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116ad036-2397-45a9-8541-2b7f05ccc721 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:31.747281 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116ad036-2397-45a9-8541-2b7f05ccc721 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:31.747745 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116ad036-2397-45a9-8541-2b7f05ccc721 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:31.765262 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c89346-4623-401d-bcc2-a0accc4b9e2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:31.766049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c89346-4623-401d-bcc2-a0accc4b9e2b None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:31.766326 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c89346-4623-401d-bcc2-a0accc4b9e2b None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:31.766596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c89346-4623-401d-bcc2-a0accc4b9e2b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:31.778140 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c89346-4623-401d-bcc2-a0accc4b9e2b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:31.782943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-f3c89346-4623-401d-bcc2-a0accc4b9e2b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:07:31.782943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:31.782943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:31.782943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:31.782943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:31.782943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:31.782943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:31.782943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:31.782943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:31.782943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:31.782943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:31.782943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:31.782943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:31.782943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:31.782943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:31.782943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:31.783884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:31.783884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:31.783884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:31.783884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:31.783884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:31.783884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:31.783884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:31.783884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:31.783884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:31.783884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:31.783884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:31.783884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:31.783884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:31.783884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:31.783884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:31.783884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:31.784895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:31.784895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:31.784895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:31.784895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:31.784895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:31.784895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:31.784895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:31.784895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:31.784895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:31.784895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:31.784895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:07:31.784895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:31.784895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 361/723] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:07:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:31.793787 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c89346-4623-401d-bcc2-a0accc4b9e2b None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:31.795643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-116ad036-2397-45a9-8541-2b7f05ccc721 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:31.798839 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-116ad036-2397-45a9-8541-2b7f05ccc721 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:31.967165 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 363/724] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:07:32.006250 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c89346-4623-401d-bcc2-a0accc4b9e2b None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:32.054351 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ab4792-7bd2-454c-8177-6458d748a4ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:32.055294 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ab4792-7bd2-454c-8177-6458d748a4ba None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:32.055783 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ab4792-7bd2-454c-8177-6458d748a4ba None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:32.056372 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ab4792-7bd2-454c-8177-6458d748a4ba None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:32.057388 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ab4792-7bd2-454c-8177-6458d748a4ba None admin] RBAC: Adding query filter params (name=admin) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:32.057943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ab4792-7bd2-454c-8177-6458d748a4ba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:32.059493 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ab4792-7bd2-454c-8177-6458d748a4ba None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:32.073204 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 362/725] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:07:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:32.078570 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116ad036-2397-45a9-8541-2b7f05ccc721 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:32.086462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:32.224678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116ad036-2397-45a9-8541-2b7f05ccc721 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:32.272176 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84fda2c-76e1-4fac-b993-d34b586218a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:32.273175 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84fda2c-76e1-4fac-b993-d34b586218a9 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:32.273663 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84fda2c-76e1-4fac-b993-d34b586218a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:32.274160 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84fda2c-76e1-4fac-b993-d34b586218a9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:32.289317 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84fda2c-76e1-4fac-b993-d34b586218a9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:32.292338 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84fda2c-76e1-4fac-b993-d34b586218a9 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:32.313547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-f84fda2c-76e1-4fac-b993-d34b586218a9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:32.313547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:32.313547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:32.313547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:32.313547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:32.313547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:32.313547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:32.313547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:32.313547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:32.313547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:32.313547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:32.313547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:32.313547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:32.313547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:32.313547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:32.313547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:32.314562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:32.314562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:32.314562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:32.314562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:32.314562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:32.314562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:32.314562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:32.314562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:32.314562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:32.314562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:32.314562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:32.314562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:32.314562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:32.314562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:32.314562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:32.314562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:32.314562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:32.315432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:32.315432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:32.315432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:32.317211 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 364/726] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:32.323365 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ab4792-7bd2-454c-8177-6458d748a4ba None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:32.344606 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:32.382175 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84fda2c-76e1-4fac-b993-d34b586218a9 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:32.441384 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ab4792-7bd2-454c-8177-6458d748a4ba None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:32.506196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:32.522476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16c9051-6c69-4b7b-b2fd-1ffb8d01c0d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:32.523091 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16c9051-6c69-4b7b-b2fd-1ffb8d01c0d8 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:32.523365 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16c9051-6c69-4b7b-b2fd-1ffb8d01c0d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:32.523635 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16c9051-6c69-4b7b-b2fd-1ffb8d01c0d8 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:32.524284 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16c9051-6c69-4b7b-b2fd-1ffb8d01c0d8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:32.524588 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16c9051-6c69-4b7b-b2fd-1ffb8d01c0d8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:32.525966 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16c9051-6c69-4b7b-b2fd-1ffb8d01c0d8 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:32.557351 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 363/727] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:32.562718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16c9051-6c69-4b7b-b2fd-1ffb8d01c0d8 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:32.583733 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84fda2c-76e1-4fac-b993-d34b586218a9 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:32.625356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0dad51-de7f-4b73-adcb-75542685778d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:32.626112 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0dad51-de7f-4b73-adcb-75542685778d None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:32.626400 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0dad51-de7f-4b73-adcb-75542685778d None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:32.626676 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0dad51-de7f-4b73-adcb-75542685778d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:32.643761 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0dad51-de7f-4b73-adcb-75542685778d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:32.650442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-4e0dad51-de7f-4b73-adcb-75542685778d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:32.650442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:32.650442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:32.650442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:32.650442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:32.650442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:32.650442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:32.650442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:32.650442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:32.650442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:32.650442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:32.650442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:32.650442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:32.650442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:32.650442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:32.650442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:32.651366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:32.651366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:32.651366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:32.651366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:32.651366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:32.651366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:32.651366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:32.651366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:32.651366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:32.651366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:32.651366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:32.651366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:32.651366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:32.651366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:32.651366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:32.651366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:32.652205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:32.652205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:32.652205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:32.652205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:32.652205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:32.652205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:32.652205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:32.652205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:32.652205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:32.652205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:32.652205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:32.652205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:32.652205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 365/728] 10.222.0.26 () {56 vars in 1102 bytes} [Tue Sep 27 04:07:32 2022] GET /identity/v3/roles/service => generated 85 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:32.653376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b855b8b9-dd89-47ff-8a21-ae9e5cc5851c None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:32.653537 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b855b8b9-dd89-47ff-8a21-ae9e5cc5851c None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:32.657770 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b855b8b9-dd89-47ff-8a21-ae9e5cc5851c None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:32.657770 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 366/729] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:32.663779 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b855b8b9-dd89-47ff-8a21-ae9e5cc5851c None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:32.668833 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:32.697580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16c9051-6c69-4b7b-b2fd-1ffb8d01c0d8 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:32.736432 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36c8a5b-07c6-4e7d-873c-61ee677e921b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:32.737055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36c8a5b-07c6-4e7d-873c-61ee677e921b None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:32.737384 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36c8a5b-07c6-4e7d-873c-61ee677e921b None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:32.737658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36c8a5b-07c6-4e7d-873c-61ee677e921b None admin] PATH_INFO: `/v3/users/swift` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:32.745683 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36c8a5b-07c6-4e7d-873c-61ee677e921b None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:32.747297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36c8a5b-07c6-4e7d-873c-61ee677e921b None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:32.761929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-e36c8a5b-07c6-4e7d-873c-61ee677e921b None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 27 04:07:32.761929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:32.761929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:32.761929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:32.761929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:32.761929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:32.761929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:32.761929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:32.761929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:32.761929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:32.761929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:32.761929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:32.761929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:32.761929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:32.761929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:32.761929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:32.762946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:32.762946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:32.762946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:32.762946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:32.762946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:32.762946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:32.762946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:32.762946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:32.762946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:32.762946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:32.762946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:32.762946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:32.762946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:32.762946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:32.762946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:32.762946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:32.762946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:32.762946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:32.763883 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:32.763883 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:32.763883 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:32.763883 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:32.763883 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:32.763883 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:32.763883 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:32.763883 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:32.763883 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 27 04:07:32.763883 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:32.763883 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 364/730] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:07:32 2022] GET /identity/v3/users/swift => generated 83 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:32.771011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef83b6b2-e523-4d48-a837-6378db538068 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:32.771011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef83b6b2-e523-4d48-a837-6378db538068 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:32.771011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef83b6b2-e523-4d48-a837-6378db538068 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:32.793651 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-ef83b6b2-e523-4d48-a837-6378db538068 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:32.794883 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ef83b6b2-e523-4d48-a837-6378db538068 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:32.812750 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b855b8b9-dd89-47ff-8a21-ae9e5cc5851c None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:32.832675 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:32.854146 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3017c233-e093-4db5-a670-674bfcc13608 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:32.855253 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3017c233-e093-4db5-a670-674bfcc13608 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:32.855733 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3017c233-e093-4db5-a670-674bfcc13608 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:32.856248 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3017c233-e093-4db5-a670-674bfcc13608 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:32.857187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3017c233-e093-4db5-a670-674bfcc13608 None admin] RBAC: Adding query filter params (name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:32.858016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3017c233-e093-4db5-a670-674bfcc13608 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:32.860307 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3017c233-e093-4db5-a670-674bfcc13608 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:32.877768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 365/731] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:07:32.877768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef83b6b2-e523-4d48-a837-6378db538068 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:32.880253 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 367/732] 10.222.0.26 () {56 vars in 1103 bytes} [Tue Sep 27 04:07:32 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:32.881822 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56879d9-fe3e-4d2a-9325-b0fa9a6c6670 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:32.882377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56879d9-fe3e-4d2a-9325-b0fa9a6c6670 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:32.882864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56879d9-fe3e-4d2a-9325-b0fa9a6c6670 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:32.884602 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 368/733] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:32.889521 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e066ce70-c094-469f-8cd7-1ef3d74fa8e0 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:32.890089 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e066ce70-c094-469f-8cd7-1ef3d74fa8e0 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:32.890573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e066ce70-c094-469f-8cd7-1ef3d74fa8e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:32.909546 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-e066ce70-c094-469f-8cd7-1ef3d74fa8e0 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:32.910865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-e066ce70-c094-469f-8cd7-1ef3d74fa8e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:32.939426 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:32.981230 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 369/734] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 04:07:32.981649 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef83b6b2-e523-4d48-a837-6378db538068 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:32.982960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae372458-c133-4d99-beb0-b4a9022cfc75 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:32.983414 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae372458-c133-4d99-beb0-b4a9022cfc75 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:32.983855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae372458-c133-4d99-beb0-b4a9022cfc75 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:32.985499 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 370/735] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:32.986364 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae372458-c133-4d99-beb0-b4a9022cfc75 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:33.032634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b2c7fb-b968-4f93-97f9-806d5d239d94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:33.033235 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b2c7fb-b968-4f93-97f9-806d5d239d94 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:33.033928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b2c7fb-b968-4f93-97f9-806d5d239d94 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:33.034208 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b2c7fb-b968-4f93-97f9-806d5d239d94 None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:33.034928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b2c7fb-b968-4f93-97f9-806d5d239d94 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:33.035231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b2c7fb-b968-4f93-97f9-806d5d239d94 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:33.037513 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b2c7fb-b968-4f93-97f9-806d5d239d94 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:33.059313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 366/736] 10.222.0.26 () {56 vars in 1135 bytes} [Tue Sep 27 04:07:32 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:33.060907 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144d09a4-c318-4338-a580-dd5381164ebc None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:33.061382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144d09a4-c318-4338-a580-dd5381164ebc None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:33.062184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144d09a4-c318-4338-a580-dd5381164ebc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:33.085678 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-144d09a4-c318-4338-a580-dd5381164ebc None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:33.087241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-144d09a4-c318-4338-a580-dd5381164ebc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:33.110994 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae372458-c133-4d99-beb0-b4a9022cfc75 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:33.123674 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:33.155388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf984f6-15b1-4e6a-9413-4221957ca711 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:33.156005 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf984f6-15b1-4e6a-9413-4221957ca711 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:33.156269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf984f6-15b1-4e6a-9413-4221957ca711 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:33.156540 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf984f6-15b1-4e6a-9413-4221957ca711 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:33.162300 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 367/737] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 04:07:33.163629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144d09a4-c318-4338-a580-dd5381164ebc None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:33.173911 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf984f6-15b1-4e6a-9413-4221957ca711 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:33.175525 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf984f6-15b1-4e6a-9413-4221957ca711 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:33.193409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-aaf984f6-15b1-4e6a-9413-4221957ca711 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:33.193409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:33.193409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:33.193409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:33.193409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:33.193409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:33.193409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:33.193409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:33.193409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:33.193409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:33.193409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:33.193409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:33.193409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:33.193409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:33.193409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:33.193409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:33.195863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:33.195863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:33.195863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:33.195863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:33.195863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:33.195863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:33.195863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:33.195863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:33.195863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:33.195863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:33.195863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:33.195863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:33.195863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:33.195863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:33.195863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:33.195863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:33.195863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:33.196783 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:33.196783 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:33.196783 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:33.198086 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 371/738] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:33.198890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf984f6-15b1-4e6a-9413-4221957ca711 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:33.228974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:33.279166 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144d09a4-c318-4338-a580-dd5381164ebc None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:33.318762 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf984f6-15b1-4e6a-9413-4221957ca711 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:33.319700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94b1e84-fba1-43fc-8c7a-10d5cb3f6ad5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:33.320460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94b1e84-fba1-43fc-8c7a-10d5cb3f6ad5 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:33.320735 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94b1e84-fba1-43fc-8c7a-10d5cb3f6ad5 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:33.321875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94b1e84-fba1-43fc-8c7a-10d5cb3f6ad5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:33.352408 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8fc241-68a8-469f-b06d-9f804b930b38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:33.353025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8fc241-68a8-469f-b06d-9f804b930b38 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:33.353302 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8fc241-68a8-469f-b06d-9f804b930b38 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:33.353571 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8fc241-68a8-469f-b06d-9f804b930b38 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:33.366087 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8fc241-68a8-469f-b06d-9f804b930b38 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:33.371377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-3c8fc241-68a8-469f-b06d-9f804b930b38 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:07:33.371377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:33.371377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:33.371377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:33.371377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:33.371377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:33.371377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:33.371377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:33.371377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:33.371377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:33.371377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:33.371377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:33.371377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:33.371377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:33.371377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:33.371377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:33.372288 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:33.372288 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:33.372288 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:33.372288 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:33.372288 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:33.372288 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:33.372288 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:33.372288 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:33.372288 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:33.372288 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:33.372288 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:33.372288 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:33.372288 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:33.372288 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:33.372288 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:33.372288 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:33.373116 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:33.373116 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:33.373116 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:33.373116 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:33.373116 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:33.373116 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:33.373116 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:33.373116 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:33.373116 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:33.373116 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:33.373116 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:07:33.373116 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:33.373116 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 372/739] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:07:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:33.373833 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8fc241-68a8-469f-b06d-9f804b930b38 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:33.398470 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:33.408651 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94b1e84-fba1-43fc-8c7a-10d5cb3f6ad5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:33.410150 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94b1e84-fba1-43fc-8c7a-10d5cb3f6ad5 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:33.464195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:33.487910 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8fc241-68a8-469f-b06d-9f804b930b38 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:33.516656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 368/740] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:07:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 04:07:33.516656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94b1e84-fba1-43fc-8c7a-10d5cb3f6ad5 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:33.522461 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8d049b-3d40-4009-828e-6fc8c109b63e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:33.523058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8d049b-3d40-4009-828e-6fc8c109b63e None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:33.523339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8d049b-3d40-4009-828e-6fc8c109b63e None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:33.523612 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8d049b-3d40-4009-828e-6fc8c109b63e None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:33.524254 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8d049b-3d40-4009-828e-6fc8c109b63e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:33.524561 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8d049b-3d40-4009-828e-6fc8c109b63e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:33.525951 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8d049b-3d40-4009-828e-6fc8c109b63e None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:33.537431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 373/741] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:33.538244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8d049b-3d40-4009-828e-6fc8c109b63e None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:33.613628 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94b1e84-fba1-43fc-8c7a-10d5cb3f6ad5 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:33.617389 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:33.637154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8d049b-3d40-4009-828e-6fc8c109b63e None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:33.647359 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3657da1-4a49-40e9-9616-466b6c9269cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:33.647955 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3657da1-4a49-40e9-9616-466b6c9269cb None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:33.648240 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3657da1-4a49-40e9-9616-466b6c9269cb None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:33.648519 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3657da1-4a49-40e9-9616-466b6c9269cb None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:33.649160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3657da1-4a49-40e9-9616-466b6c9269cb None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:33.650552 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3657da1-4a49-40e9-9616-466b6c9269cb None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:33.674183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 369/742] 10.222.0.26 () {60 vars in 1141 bytes} [Tue Sep 27 04:07:33 2022] POST /identity/v3/services => generated 251 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:07:33.674993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3657da1-4a49-40e9-9616-466b6c9269cb None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:33.685339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7dc40c-0576-42f3-b03d-2a469c151c87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:33.685988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7dc40c-0576-42f3-b03d-2a469c151c87 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:33.686270 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7dc40c-0576-42f3-b03d-2a469c151c87 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:33.686535 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7dc40c-0576-42f3-b03d-2a469c151c87 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:33.687215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7dc40c-0576-42f3-b03d-2a469c151c87 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:33.687534 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7dc40c-0576-42f3-b03d-2a469c151c87 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:33.689006 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7dc40c-0576-42f3-b03d-2a469c151c87 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:33.694491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:33.700403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 374/743] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:07:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:33.701289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7dc40c-0576-42f3-b03d-2a469c151c87 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:33.797177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3657da1-4a49-40e9-9616-466b6c9269cb None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:33.829725 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7dc40c-0576-42f3-b03d-2a469c151c87 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:33.836155 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7146eb-d45c-47ee-b2cf-9165c983b71c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:33.838359 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7146eb-d45c-47ee-b2cf-9165c983b71c None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:33.838359 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7146eb-d45c-47ee-b2cf-9165c983b71c None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:33.838567 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7146eb-d45c-47ee-b2cf-9165c983b71c None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:33.839095 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7146eb-d45c-47ee-b2cf-9165c983b71c None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:33.840809 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7146eb-d45c-47ee-b2cf-9165c983b71c None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:33.857383 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 370/744] 10.222.0.26 () {60 vars in 1141 bytes} [Tue Sep 27 04:07:33 2022] POST /identity/v3/services => generated 240 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:07:33.858810 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7146eb-d45c-47ee-b2cf-9165c983b71c None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:33.885465 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5bf3c6-99ef-4670-ad86-175df7717d68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:33.886344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5bf3c6-99ef-4670-ad86-175df7717d68 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:33.886622 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5bf3c6-99ef-4670-ad86-175df7717d68 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:33.887565 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5bf3c6-99ef-4670-ad86-175df7717d68 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:33.891415 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:33.904100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5bf3c6-99ef-4670-ad86-175df7717d68 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:33.905571 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5bf3c6-99ef-4670-ad86-175df7717d68 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:33.925085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-3b5bf3c6-99ef-4670-ad86-175df7717d68 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:33.925085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:33.925085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:33.925085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:33.925085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:33.925085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:33.925085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:33.925085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:33.925085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:33.925085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:33.925085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:33.925085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:33.925085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:33.925085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:33.925085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:33.925085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:33.926233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:33.926233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:33.926233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:33.926233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:33.926233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:33.926233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:33.926233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:33.926233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:33.926233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:33.926233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:33.926233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:33.926233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:33.926233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:33.926233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:33.926233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:33.926233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:33.926233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:33.927702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:33.927702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:33.927702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:33.927702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 375/745] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:33.927702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5bf3c6-99ef-4670-ad86-175df7717d68 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:33.955346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:34.003639 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7146eb-d45c-47ee-b2cf-9165c983b71c None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:34.053272 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746157a4-cd5b-4d20-beb6-8a0ceb595c3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:34.054089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746157a4-cd5b-4d20-beb6-8a0ceb595c3f None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:34.054364 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746157a4-cd5b-4d20-beb6-8a0ceb595c3f None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:34.054756 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746157a4-cd5b-4d20-beb6-8a0ceb595c3f None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:34.064620 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746157a4-cd5b-4d20-beb6-8a0ceb595c3f None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:34.066370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746157a4-cd5b-4d20-beb6-8a0ceb595c3f None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:34.072462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5bf3c6-99ef-4670-ad86-175df7717d68 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:34.081303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-746157a4-cd5b-4d20-beb6-8a0ceb595c3f None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 27 04:07:34.081303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:34.081303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:34.081303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:34.081303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:34.081303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:34.081303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:34.081303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:34.081303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:34.081303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:34.081303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:34.081303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:34.081303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:34.081303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:34.081303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:34.081303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:34.082465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:34.082465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:34.082465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:34.082465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:34.082465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:34.082465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:34.082465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:34.082465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:34.082465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:34.082465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:34.082465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:34.082465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:34.082465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:34.082465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:34.082465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:34.082465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:34.082465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:34.082465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:34.083998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:34.083998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:34.083998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:34.083998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:34.083998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:34.083998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:34.083998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:34.083998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:34.083998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 27 04:07:34.083998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:34.083998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 371/746] 10.222.0.26 () {56 vars in 1117 bytes} [Tue Sep 27 04:07:33 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:34.084585 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746157a4-cd5b-4d20-beb6-8a0ceb595c3f None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:34.119218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4bbcfc-72dd-4014-80c6-75c1d5265456 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:34.119885 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4bbcfc-72dd-4014-80c6-75c1d5265456 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:34.120264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4bbcfc-72dd-4014-80c6-75c1d5265456 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:34.120539 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4bbcfc-72dd-4014-80c6-75c1d5265456 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:34.132819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4bbcfc-72dd-4014-80c6-75c1d5265456 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:34.134312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4bbcfc-72dd-4014-80c6-75c1d5265456 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:34.150918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-fd4bbcfc-72dd-4014-80c6-75c1d5265456 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:34.150918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:34.150918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:34.150918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:34.150918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:34.150918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:34.150918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:34.150918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:34.150918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:34.150918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:34.150918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:34.150918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:34.150918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:34.150918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:34.150918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:34.150918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:34.153125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:34.153125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:34.153125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:34.153125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:34.153125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:34.153125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:34.153125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:34.153125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:34.153125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:34.153125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:34.153125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:34.153125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:34.153125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:34.153125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:34.153125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:34.153125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:34.153125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:34.155470 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:34.155470 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:34.155470 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:34.155470 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 376/747] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:34.155470 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4bbcfc-72dd-4014-80c6-75c1d5265456 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:34.200075 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:34.200954 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746157a4-cd5b-4d20-beb6-8a0ceb595c3f None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:34.239374 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:34.246604 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b89926-b3eb-42c5-bad4-e0ddee0b2db0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:34.247475 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b89926-b3eb-42c5-bad4-e0ddee0b2db0 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:34.247751 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b89926-b3eb-42c5-bad4-e0ddee0b2db0 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:34.248021 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b89926-b3eb-42c5-bad4-e0ddee0b2db0 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:34.248923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b89926-b3eb-42c5-bad4-e0ddee0b2db0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:34.249218 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b89926-b3eb-42c5-bad4-e0ddee0b2db0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:34.251382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b89926-b3eb-42c5-bad4-e0ddee0b2db0 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:34.268718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 372/748] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:34.269131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4bbcfc-72dd-4014-80c6-75c1d5265456 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:34.269614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b89926-b3eb-42c5-bad4-e0ddee0b2db0 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:34.306423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9889c6-8e40-4a7e-8943-c8916dea735e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:34.307409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9889c6-8e40-4a7e-8943-c8916dea735e None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:34.307902 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9889c6-8e40-4a7e-8943-c8916dea735e None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:34.308406 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9889c6-8e40-4a7e-8943-c8916dea735e None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:34.309329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9889c6-8e40-4a7e-8943-c8916dea735e None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:34.309879 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9889c6-8e40-4a7e-8943-c8916dea735e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:34.311576 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9889c6-8e40-4a7e-8943-c8916dea735e None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:34.329846 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 377/749] 10.222.0.26 () {56 vars in 1149 bytes} [Tue Sep 27 04:07:34 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:34.331717 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9889c6-8e40-4a7e-8943-c8916dea735e None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:34.402442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b89926-b3eb-42c5-bad4-e0ddee0b2db0 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:34.444886 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ae2707-b50d-406c-97d0-e03a77b3249d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:34.445654 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ae2707-b50d-406c-97d0-e03a77b3249d None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:34.446056 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ae2707-b50d-406c-97d0-e03a77b3249d None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:34.446430 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ae2707-b50d-406c-97d0-e03a77b3249d None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:34.447267 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ae2707-b50d-406c-97d0-e03a77b3249d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:34.447669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ae2707-b50d-406c-97d0-e03a77b3249d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:34.449293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ae2707-b50d-406c-97d0-e03a77b3249d None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:34.451522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:34.459388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9889c6-8e40-4a7e-8943-c8916dea735e None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:34.464493 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 373/750] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:34.465504 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ae2707-b50d-406c-97d0-e03a77b3249d None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:34.510174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf09aa6a-3bb1-4190-9178-7d5325caa018 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:34.511416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf09aa6a-3bb1-4190-9178-7d5325caa018 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:34.511793 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf09aa6a-3bb1-4190-9178-7d5325caa018 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:34.512195 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf09aa6a-3bb1-4190-9178-7d5325caa018 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:34.522270 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf09aa6a-3bb1-4190-9178-7d5325caa018 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:34.524014 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:34.525136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf09aa6a-3bb1-4190-9178-7d5325caa018 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:34.543647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-cf09aa6a-3bb1-4190-9178-7d5325caa018 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:34.543647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:34.543647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:34.543647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:34.543647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:34.543647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:34.543647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:34.543647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:34.543647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:34.543647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:34.543647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:34.543647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:34.543647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:34.543647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:34.543647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:34.544648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:34.544648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:34.544648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:34.544648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:34.544648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:34.544648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:34.544648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:34.544648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:34.544648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:34.544648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:34.544648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:34.544648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:34.544648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:34.544648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:34.544648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:34.544648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:34.544648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:34.545564 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:34.545564 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:34.545564 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:34.545564 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:34.545564 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:34.545564 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:34.545564 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:34.545564 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:34.545564 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 378/751] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:34.546751 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf09aa6a-3bb1-4190-9178-7d5325caa018 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:34.579239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ae2707-b50d-406c-97d0-e03a77b3249d None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:34.621636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a020f0-ce9c-4730-b2f8-ec01a641bc71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:34.622657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a020f0-ce9c-4730-b2f8-ec01a641bc71 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:34.623151 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a020f0-ce9c-4730-b2f8-ec01a641bc71 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:34.623624 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a020f0-ce9c-4730-b2f8-ec01a641bc71 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:34.681872 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf09aa6a-3bb1-4190-9178-7d5325caa018 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:34.701046 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a020f0-ce9c-4730-b2f8-ec01a641bc71 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:34.702961 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a020f0-ce9c-4730-b2f8-ec01a641bc71 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:34.731550 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b70c60-3625-4cb4-a6e6-34cc4a028c63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:34.732197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b70c60-3625-4cb4-a6e6-34cc4a028c63 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:34.732699 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b70c60-3625-4cb4-a6e6-34cc4a028c63 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:34.732976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b70c60-3625-4cb4-a6e6-34cc4a028c63 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:34.742216 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b70c60-3625-4cb4-a6e6-34cc4a028c63 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:34.744147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b70c60-3625-4cb4-a6e6-34cc4a028c63 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:34.758318 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:34.760423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-92b70c60-3625-4cb4-a6e6-34cc4a028c63 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 27 04:07:34.760423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:34.760423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:34.760423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:34.760423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:34.760423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:34.760423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:34.760423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:34.760423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:34.760423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:34.760423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:34.760423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:34.760423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:34.760423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:34.760423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:34.760423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:34.761851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:34.761851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:34.761851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:34.761851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:34.761851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:34.761851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:34.761851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:34.761851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:34.761851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:34.761851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:34.761851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:34.761851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:34.761851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:34.761851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:34.761851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:34.761851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:34.761851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:34.761851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:34.762860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:34.762860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:34.762860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:34.762860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:34.762860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:34.762860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:34.762860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:34.762860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:34.762860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 27 04:07:34.762860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:34.762860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 379/752] 10.222.0.26 () {56 vars in 1093 bytes} [Tue Sep 27 04:07:34 2022] GET /identity/v3/users/nova => generated 82 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:34.763452 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b70c60-3625-4cb4-a6e6-34cc4a028c63 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:34.785349 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:34.829705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 374/753] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:07:34 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 04:07:34.830763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a020f0-ce9c-4730-b2f8-ec01a641bc71 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:34.884908 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b70c60-3625-4cb4-a6e6-34cc4a028c63 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:34.919164 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996e1b0a-17a2-4c6c-a861-2f387c908b23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:34.919989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996e1b0a-17a2-4c6c-a861-2f387c908b23 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:34.920511 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996e1b0a-17a2-4c6c-a861-2f387c908b23 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:34.920966 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996e1b0a-17a2-4c6c-a861-2f387c908b23 None admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:34.921965 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996e1b0a-17a2-4c6c-a861-2f387c908b23 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:34.922478 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996e1b0a-17a2-4c6c-a861-2f387c908b23 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:34.924098 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996e1b0a-17a2-4c6c-a861-2f387c908b23 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:34.936831 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a020f0-ce9c-4730-b2f8-ec01a641bc71 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:34.937663 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 380/754] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:34.938720 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996e1b0a-17a2-4c6c-a861-2f387c908b23 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:34.973600 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550368b5-3529-47df-806c-b1a655269ed4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:34.974971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550368b5-3529-47df-806c-b1a655269ed4 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:34.974971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550368b5-3529-47df-806c-b1a655269ed4 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:34.974971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550368b5-3529-47df-806c-b1a655269ed4 None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:34.975568 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550368b5-3529-47df-806c-b1a655269ed4 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:34.975871 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550368b5-3529-47df-806c-b1a655269ed4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:34.977284 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550368b5-3529-47df-806c-b1a655269ed4 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:34.995874 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 375/755] 10.222.0.26 () {56 vars in 1133 bytes} [Tue Sep 27 04:07:34 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:34.996832 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550368b5-3529-47df-806c-b1a655269ed4 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:35.010352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:35.020251 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:35.043304 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996e1b0a-17a2-4c6c-a861-2f387c908b23 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:35.080735 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39fe930b-f67c-4be9-9716-de59c5ec032f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:35.082202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fe930b-f67c-4be9-9716-de59c5ec032f None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:35.082774 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fe930b-f67c-4be9-9716-de59c5ec032f None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:35.083281 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fe930b-f67c-4be9-9716-de59c5ec032f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:35.095036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39fe930b-f67c-4be9-9716-de59c5ec032f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:35.096708 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39fe930b-f67c-4be9-9716-de59c5ec032f None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:35.105635 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550368b5-3529-47df-806c-b1a655269ed4 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:35.115293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-39fe930b-f67c-4be9-9716-de59c5ec032f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:35.115293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:35.115293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:35.115293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:35.115293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:35.115293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:35.115293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:35.115293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:35.115293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:35.115293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:35.115293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:35.115293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:35.115293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:35.115293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:35.115293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:35.115293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:35.116339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:35.116339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:35.116339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:35.116339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:35.116339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:35.116339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:35.116339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:35.116339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:35.116339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:35.116339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:35.116339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:35.116339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:35.116339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:35.116339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:35.116339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:35.116339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:35.116339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:35.117287 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:35.117287 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:35.117287 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:35.119642 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 381/756] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:35.121283 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39fe930b-f67c-4be9-9716-de59c5ec032f None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:35.150381 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c12347-0bb4-4d0a-8392-ac5acda4edd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:35.151115 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c12347-0bb4-4d0a-8392-ac5acda4edd1 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:35.151484 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c12347-0bb4-4d0a-8392-ac5acda4edd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:35.151849 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c12347-0bb4-4d0a-8392-ac5acda4edd1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:35.152666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c12347-0bb4-4d0a-8392-ac5acda4edd1 None admin] RBAC: Adding query filter params (role.id=e6a31562a18649f0809202c36a8e18ef, scope.project.id=c830a3bfece6456486f4b094076b5364, user.id=181dd5dc720c4a97860852062b7b19c3) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:35.153062 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c12347-0bb4-4d0a-8392-ac5acda4edd1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:35.154699 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c12347-0bb4-4d0a-8392-ac5acda4edd1 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:35.160627 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 376/757] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:07:34 2022] GET /identity/v3/role_assignments?user.id=181dd5dc720c4a97860852062b7b19c3&scope.project.id=c830a3bfece6456486f4b094076b5364&role.id=e6a31562a18649f0809202c36a8e18ef => generated 259 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:35.161591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c12347-0bb4-4d0a-8392-ac5acda4edd1 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:35.247126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:35.250006 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39fe930b-f67c-4be9-9716-de59c5ec032f None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:35.290039 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:35.306996 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c12347-0bb4-4d0a-8392-ac5acda4edd1 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:35.309989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21656d2b-bcbe-4bd0-ae74-7013d703216d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:35.310684 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21656d2b-bcbe-4bd0-ae74-7013d703216d None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:35.310964 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21656d2b-bcbe-4bd0-ae74-7013d703216d None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:35.311231 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21656d2b-bcbe-4bd0-ae74-7013d703216d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:35.335909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21656d2b-bcbe-4bd0-ae74-7013d703216d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:35.337408 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21656d2b-bcbe-4bd0-ae74-7013d703216d None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:35.368197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-21656d2b-bcbe-4bd0-ae74-7013d703216d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:35.368197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:35.368197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:35.368197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:35.368197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:35.368197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:35.368197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:35.368197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:35.368197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:35.368197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:35.368197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:35.368197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:35.368197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:35.368197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:35.368197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:35.368197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:35.369233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:35.369233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:35.369233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:35.369233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:35.369233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:35.369233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:35.369233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:35.369233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:35.369233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:35.369233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:35.369233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:35.369233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:35.369233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:35.369233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:35.369233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:35.369233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:35.369233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:35.370160 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:35.370160 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:35.370160 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:35.370160 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 382/758] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:35.375131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21656d2b-bcbe-4bd0-ae74-7013d703216d None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:35.388340 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a0b7af-62f2-4841-9f89-418cba8e31b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:35.388946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a0b7af-62f2-4841-9f89-418cba8e31b7 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:35.389219 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a0b7af-62f2-4841-9f89-418cba8e31b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:35.389489 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a0b7af-62f2-4841-9f89-418cba8e31b7 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:35.390158 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a0b7af-62f2-4841-9f89-418cba8e31b7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:35.390458 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a0b7af-62f2-4841-9f89-418cba8e31b7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:35.391804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a0b7af-62f2-4841-9f89-418cba8e31b7 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:35.405010 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 377/759] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:35.410828 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a0b7af-62f2-4841-9f89-418cba8e31b7 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:35.530347 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21656d2b-bcbe-4bd0-ae74-7013d703216d None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:35.570135 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b02813d-9804-42d8-8d4e-3a4b98fe851e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:35.570750 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b02813d-9804-42d8-8d4e-3a4b98fe851e None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:35.571031 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b02813d-9804-42d8-8d4e-3a4b98fe851e None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:35.571304 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b02813d-9804-42d8-8d4e-3a4b98fe851e None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:35.571958 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b02813d-9804-42d8-8d4e-3a4b98fe851e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:35.572259 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b02813d-9804-42d8-8d4e-3a4b98fe851e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:35.573615 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b02813d-9804-42d8-8d4e-3a4b98fe851e None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:35.584421 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a0b7af-62f2-4841-9f89-418cba8e31b7 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:35.588066 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 383/760] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:35.594481 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b02813d-9804-42d8-8d4e-3a4b98fe851e None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:35.646728 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:35.659478 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a877da-5c32-49fd-8c83-ece348a47e6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:35.660104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a877da-5c32-49fd-8c83-ece348a47e6e None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:35.660380 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a877da-5c32-49fd-8c83-ece348a47e6e None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:35.660643 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a877da-5c32-49fd-8c83-ece348a47e6e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:35.663555 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:35.669659 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a877da-5c32-49fd-8c83-ece348a47e6e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:35.671579 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a877da-5c32-49fd-8c83-ece348a47e6e None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:35.689179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-f9a877da-5c32-49fd-8c83-ece348a47e6e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:35.689179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:35.689179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:35.689179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:35.689179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:35.689179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:35.689179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:35.689179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:35.689179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:35.689179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:35.689179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:35.689179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:35.689179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:35.689179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:35.689179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:35.690160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:35.690160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:35.690160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:35.690160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:35.690160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:35.690160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:35.690160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:35.690160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:35.690160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:35.690160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:35.690160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:35.690160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:35.690160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:35.690160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:35.690160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:35.690160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:35.690160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:35.691056 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:35.691056 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:35.691056 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:35.691056 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:35.691056 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:35.691056 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:35.691056 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:35.691056 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:35.696433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 378/761] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:07:35.697552 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a877da-5c32-49fd-8c83-ece348a47e6e None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:35.721159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b02813d-9804-42d8-8d4e-3a4b98fe851e None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:35.788957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37fa6398-1240-46b3-b74f-0305f07062a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:35.789921 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fa6398-1240-46b3-b74f-0305f07062a0 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:35.790383 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fa6398-1240-46b3-b74f-0305f07062a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:35.790664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fa6398-1240-46b3-b74f-0305f07062a0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:35.804958 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37fa6398-1240-46b3-b74f-0305f07062a0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:35.813777 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37fa6398-1240-46b3-b74f-0305f07062a0 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:35.822580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-37fa6398-1240-46b3-b74f-0305f07062a0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:35.822580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:35.822580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:35.822580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:35.822580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:35.822580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:35.822580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:35.822580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:35.822580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:35.822580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:35.822580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:35.822580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:35.822580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:35.822580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:35.822580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:35.823485 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:35.823485 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:35.823485 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:35.823485 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:35.823485 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:35.823485 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:35.823485 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:35.823485 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:35.823485 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:35.823485 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:35.823485 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:35.823485 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:35.823485 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:35.823485 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:35.823485 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:35.823485 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:35.823485 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:35.824932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:35.824932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:35.824932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:35.824932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:35.824932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:35.824932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:35.824932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:35.824932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:35.824932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 384/762] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:07:35.830744 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37fa6398-1240-46b3-b74f-0305f07062a0 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:35.871369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a877da-5c32-49fd-8c83-ece348a47e6e None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:35.937807 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2be5c7e-f8c3-499c-a5b6-2e9a15d63e19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:35.938415 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2be5c7e-f8c3-499c-a5b6-2e9a15d63e19 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:35.938692 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2be5c7e-f8c3-499c-a5b6-2e9a15d63e19 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:35.938963 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2be5c7e-f8c3-499c-a5b6-2e9a15d63e19 None admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:35.939655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2be5c7e-f8c3-499c-a5b6-2e9a15d63e19 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:35.939958 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2be5c7e-f8c3-499c-a5b6-2e9a15d63e19 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:35.941379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2be5c7e-f8c3-499c-a5b6-2e9a15d63e19 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:35.965734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 379/763] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:07:35.972390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2be5c7e-f8c3-499c-a5b6-2e9a15d63e19 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:36.002898 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:36.008185 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37fa6398-1240-46b3-b74f-0305f07062a0 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:36.078291 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae5e09b-3b4b-4fc0-9e79-c017095cb256 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:36.078837 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae5e09b-3b4b-4fc0-9e79-c017095cb256 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:36.079118 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae5e09b-3b4b-4fc0-9e79-c017095cb256 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:36.079394 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae5e09b-3b4b-4fc0-9e79-c017095cb256 None admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:36.080106 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae5e09b-3b4b-4fc0-9e79-c017095cb256 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:36.080416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae5e09b-3b4b-4fc0-9e79-c017095cb256 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:36.081922 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae5e09b-3b4b-4fc0-9e79-c017095cb256 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:36.087609 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:36.115966 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 385/764] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:36.125064 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae5e09b-3b4b-4fc0-9e79-c017095cb256 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:36.186678 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2be5c7e-f8c3-499c-a5b6-2e9a15d63e19 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:36.226757 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3337876c-5f3f-4700-9afb-c29decb7e14a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:36.227758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3337876c-5f3f-4700-9afb-c29decb7e14a None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:36.228226 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3337876c-5f3f-4700-9afb-c29decb7e14a None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:36.228872 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3337876c-5f3f-4700-9afb-c29decb7e14a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:36.229808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3337876c-5f3f-4700-9afb-c29decb7e14a None admin] RBAC: Adding query filter params (role.id=20d83a8838004f60a32f8427577c7c35, scope.project.id=c830a3bfece6456486f4b094076b5364, user.id=5a2fcc8c9ac44dceadcc30d163bbfa0e) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:36.230308 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3337876c-5f3f-4700-9afb-c29decb7e14a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:36.232048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3337876c-5f3f-4700-9afb-c29decb7e14a None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:36.245891 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 380/765] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:07:35 2022] GET /identity/v3/role_assignments?user.id=5a2fcc8c9ac44dceadcc30d163bbfa0e&scope.project.id=c830a3bfece6456486f4b094076b5364&role.id=20d83a8838004f60a32f8427577c7c35 => generated 259 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:36.247719 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974d38bd-e61f-498b-9ffd-61a6cfe0dd48 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:36.248237 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974d38bd-e61f-498b-9ffd-61a6cfe0dd48 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:36.248707 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974d38bd-e61f-498b-9ffd-61a6cfe0dd48 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:36.250689 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 381/766] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:36.256864 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a3c106-e25c-4a60-89d1-ca69a411efc9 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:36.257457 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a3c106-e25c-4a60-89d1-ca69a411efc9 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:36.257971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a3c106-e25c-4a60-89d1-ca69a411efc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:36.279085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-c5a3c106-e25c-4a60-89d1-ca69a411efc9 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:36.279574 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae5e09b-3b4b-4fc0-9e79-c017095cb256 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:36.280283 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c5a3c106-e25c-4a60-89d1-ca69a411efc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:36.326996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91d4086-fe60-4901-889c-26027bfa888c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:36.327666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91d4086-fe60-4901-889c-26027bfa888c None admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:36.327945 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91d4086-fe60-4901-889c-26027bfa888c None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:36.328249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91d4086-fe60-4901-889c-26027bfa888c None admin] PATH_INFO: `/v3/projects/c830a3bfece6456486f4b094076b5364/users/a1586ee841cd44658e4d5feb25de7940/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:36.330960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e91d4086-fe60-4901-889c-26027bfa888c None admin] RBAC: Authorizing `identity:create_grant(project_id=c830a3bfece6456486f4b094076b5364, user_id=a1586ee841cd44658e4d5feb25de7940, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:36.333352 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e91d4086-fe60-4901-889c-26027bfa888c None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:36.355074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 386/767] 10.222.0.26 () {58 vars in 1439 bytes} [Tue Sep 27 04:07:36 2022] PUT /identity/v3/projects/c830a3bfece6456486f4b094076b5364/users/a1586ee841cd44658e4d5feb25de7940/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:07:36.363873 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5157d5c9-2fd6-475c-86cf-eaa4c155603f None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:36.364790 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5157d5c9-2fd6-475c-86cf-eaa4c155603f None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:36.366615 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5157d5c9-2fd6-475c-86cf-eaa4c155603f None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:36.377558 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 387/768] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:36 2022] GET /identity => generated 270 bytes in 16 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:36.380955 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6ad151-c129-494a-ba72-cf0a09d831bf None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:36.381249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6ad151-c129-494a-ba72-cf0a09d831bf None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:36.381492 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6ad151-c129-494a-ba72-cf0a09d831bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:36.390072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 382/769] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:36 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 04:07:36.421718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-da6ad151-c129-494a-ba72-cf0a09d831bf None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:36.423117 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-da6ad151-c129-494a-ba72-cf0a09d831bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:36.453064 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:36.516474 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 388/770] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:36 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:07:36.695760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a3c106-e25c-4a60-89d1-ca69a411efc9 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:36.748861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:36.801355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a3c106-e25c-4a60-89d1-ca69a411efc9 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:36.835879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d38de1-7a85-405e-8792-84e2bd9c3e6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:36.836512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d38de1-7a85-405e-8792-84e2bd9c3e6d None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:36.836789 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d38de1-7a85-405e-8792-84e2bd9c3e6d None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:36.837058 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d38de1-7a85-405e-8792-84e2bd9c3e6d None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:36.837658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d38de1-7a85-405e-8792-84e2bd9c3e6d None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:36.839087 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d38de1-7a85-405e-8792-84e2bd9c3e6d None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:36.847984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-05d38de1-7a85-405e-8792-84e2bd9c3e6d None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 27 04:07:36.847984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:36.847984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:07:36.847984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:07:36.847984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:07:36.847984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:07:36.847984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:07:36.847984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:36.847984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 27 04:07:36.847984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:36.847984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:07:36.847984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:36.847984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:36.847984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:36.847984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:36.847984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:36.847984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:36.849509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:36.849509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:36.849509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:36.849509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:36.849509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:36.849509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:36.849509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:07:36.849509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:07:36.849509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:07:36.849509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:07:36.849509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:36.849509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:36.849509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:36.849509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:36.849509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:36.849509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:36.850960 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:36.850960 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:36.850960 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:36.850960 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:36.850960 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:36.850960 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:36.850960 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:36.850960 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:36.850960 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:36.850960 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:36.850960 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:07:36.850960 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:36.850960 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 27 04:07:36.850960 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:36.850960 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 383/771] 10.222.0.26 () {56 vars in 1129 bytes} [Tue Sep 27 04:07:36 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:36.855680 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6ad151-c129-494a-ba72-cf0a09d831bf None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:36.864998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d38de1-7a85-405e-8792-84e2bd9c3e6d None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:37.031969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6ad151-c129-494a-ba72-cf0a09d831bf None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:37.048175 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d38de1-7a85-405e-8792-84e2bd9c3e6d None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:37.073526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:37.106096 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1742f745-efa2-473b-907f-477e3bbf4c47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:37.106705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1742f745-efa2-473b-907f-477e3bbf4c47 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:37.106987 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1742f745-efa2-473b-907f-477e3bbf4c47 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:37.107262 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1742f745-efa2-473b-907f-477e3bbf4c47 None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:37.107894 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1742f745-efa2-473b-907f-477e3bbf4c47 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:37.108195 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1742f745-efa2-473b-907f-477e3bbf4c47 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:37.109542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1742f745-efa2-473b-907f-477e3bbf4c47 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:37.112324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194c6e9e-bef7-43c7-95e5-c75210dd036c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:37.112947 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194c6e9e-bef7-43c7-95e5-c75210dd036c None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:37.113220 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194c6e9e-bef7-43c7-95e5-c75210dd036c None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:37.113488 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194c6e9e-bef7-43c7-95e5-c75210dd036c None admin] PATH_INFO: `/v3/services/network` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:37.114095 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194c6e9e-bef7-43c7-95e5-c75210dd036c None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:37.115672 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194c6e9e-bef7-43c7-95e5-c75210dd036c None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:37.122649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 389/772] 10.222.0.26 () {56 vars in 1124 bytes} [Tue Sep 27 04:07:36 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:37.127706 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1742f745-efa2-473b-907f-477e3bbf4c47 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:37.129638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-194c6e9e-bef7-43c7-95e5-c75210dd036c None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 27 04:07:37.129638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:37.129638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:07:37.129638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:07:37.129638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:07:37.129638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:07:37.129638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:07:37.129638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:37.129638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 27 04:07:37.129638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:37.129638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:07:37.129638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:37.129638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:37.129638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:37.129638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:37.129638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:37.129638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:37.130700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:37.130700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:37.130700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:37.130700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:37.130700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:37.130700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:37.130700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:07:37.130700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:07:37.130700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:07:37.130700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:07:37.130700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:37.130700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:37.130700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:37.130700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:37.130700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:37.130700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:37.131539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:37.131539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:37.131539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:37.131539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:37.131539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:37.131539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:37.131539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:37.131539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:37.131539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:37.131539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:37.131539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:07:37.131539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:37.131539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 27 04:07:37.131539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:37.138727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 384/773] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:36 2022] GET /identity/v3/services/network => generated 88 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:07:37.143700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194c6e9e-bef7-43c7-95e5-c75210dd036c None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:37.162886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:37.296799 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194c6e9e-bef7-43c7-95e5-c75210dd036c None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:37.336829 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1742f745-efa2-473b-907f-477e3bbf4c47 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:37.363815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf0e4b6-f07f-4d7a-a2ed-5dd3859a9b73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:37.364451 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf0e4b6-f07f-4d7a-a2ed-5dd3859a9b73 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:37.364728 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf0e4b6-f07f-4d7a-a2ed-5dd3859a9b73 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:37.364996 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf0e4b6-f07f-4d7a-a2ed-5dd3859a9b73 None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:37.365626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf0e4b6-f07f-4d7a-a2ed-5dd3859a9b73 None admin] RBAC: Adding query filter params (name=network) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:37.366057 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf0e4b6-f07f-4d7a-a2ed-5dd3859a9b73 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:37.367448 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf0e4b6-f07f-4d7a-a2ed-5dd3859a9b73 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:37.379937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 385/774] 10.222.0.26 () {56 vars in 1112 bytes} [Tue Sep 27 04:07:37 2022] GET /identity/v3/services?name=network => generated 124 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:37.385117 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf0e4b6-f07f-4d7a-a2ed-5dd3859a9b73 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:37.409592 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c3b0c9-23de-4fb8-b2c0-61242cfe11b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:37.410229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c3b0c9-23de-4fb8-b2c0-61242cfe11b4 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:37.410503 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c3b0c9-23de-4fb8-b2c0-61242cfe11b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:37.410769 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c3b0c9-23de-4fb8-b2c0-61242cfe11b4 None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:37.412028 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c3b0c9-23de-4fb8-b2c0-61242cfe11b4 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:37.412329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c3b0c9-23de-4fb8-b2c0-61242cfe11b4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:37.413676 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c3b0c9-23de-4fb8-b2c0-61242cfe11b4 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:37.422512 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 390/775] 10.222.0.26 () {56 vars in 1124 bytes} [Tue Sep 27 04:07:37 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:37.427950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c3b0c9-23de-4fb8-b2c0-61242cfe11b4 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:37.477044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:37.539500 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf0e4b6-f07f-4d7a-a2ed-5dd3859a9b73 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:37.602186 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:37.613877 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91510392-7921-4fd0-a9d9-1f90168f90ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:37.618094 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91510392-7921-4fd0-a9d9-1f90168f90ec None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:37.618419 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91510392-7921-4fd0-a9d9-1f90168f90ec None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:37.618682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91510392-7921-4fd0-a9d9-1f90168f90ec None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:37.619365 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91510392-7921-4fd0-a9d9-1f90168f90ec None admin] RBAC: Adding query filter params (type=network) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:37.619665 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91510392-7921-4fd0-a9d9-1f90168f90ec None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:37.621084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91510392-7921-4fd0-a9d9-1f90168f90ec None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:37.634183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 386/776] 10.222.0.26 () {56 vars in 1112 bytes} [Tue Sep 27 04:07:37 2022] GET /identity/v3/services?type=network => generated 350 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:07:37.639685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91510392-7921-4fd0-a9d9-1f90168f90ec None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:37.653427 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c3b0c9-23de-4fb8-b2c0-61242cfe11b4 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:37.721232 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3007bf-3ddc-400f-9e6c-7ef1342a41ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:37.721879 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3007bf-3ddc-400f-9e6c-7ef1342a41ee None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:37.722136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3007bf-3ddc-400f-9e6c-7ef1342a41ee None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:37.722405 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3007bf-3ddc-400f-9e6c-7ef1342a41ee None admin] PATH_INFO: `/v3/endpoints` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:37.723101 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3007bf-3ddc-400f-9e6c-7ef1342a41ee None admin] RBAC: Adding query filter params (interface=public, service_id=b3d003472e994b09ae5b59df9848002d, region_id=RegionOne) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:37.723404 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3007bf-3ddc-400f-9e6c-7ef1342a41ee None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:37.725993 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3007bf-3ddc-400f-9e6c-7ef1342a41ee None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:37.737002 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 391/777] 10.222.0.26 () {56 vars in 1251 bytes} [Tue Sep 27 04:07:37 2022] GET /identity/v3/endpoints?service_id=b3d003472e994b09ae5b59df9848002d&interface=public®ion_id=RegionOne => generated 194 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:37.742380 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3007bf-3ddc-400f-9e6c-7ef1342a41ee None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:37.799456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91510392-7921-4fd0-a9d9-1f90168f90ec None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:37.803005 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:37.839016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59329f9-6ad9-44da-b68e-508b1ce0b559 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:37.839624 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59329f9-6ad9-44da-b68e-508b1ce0b559 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:37.839904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59329f9-6ad9-44da-b68e-508b1ce0b559 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:37.840171 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59329f9-6ad9-44da-b68e-508b1ce0b559 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:37.840966 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59329f9-6ad9-44da-b68e-508b1ce0b559 None admin] RBAC: Adding query filter params (interface=public, service_id=f985712f6b634c968a1bce23971a4678, region_id=RegionOne) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:37.841272 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59329f9-6ad9-44da-b68e-508b1ce0b559 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:37.842776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59329f9-6ad9-44da-b68e-508b1ce0b559 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:37.851031 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 387/778] 10.222.0.26 () {56 vars in 1251 bytes} [Tue Sep 27 04:07:37 2022] GET /identity/v3/endpoints?service_id=f985712f6b634c968a1bce23971a4678&interface=public®ion_id=RegionOne => generated 194 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:37.852252 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d900f0f-f48f-40d6-a497-73b251651682 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:37.852477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d900f0f-f48f-40d6-a497-73b251651682 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:37.852750 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d900f0f-f48f-40d6-a497-73b251651682 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:37.854399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 388/779] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:37.859326 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29d1d37-a7ed-4e53-a92c-ba46caabc70e None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:37.859616 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29d1d37-a7ed-4e53-a92c-ba46caabc70e None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:37.859844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3007bf-3ddc-400f-9e6c-7ef1342a41ee None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:37.859952 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29d1d37-a7ed-4e53-a92c-ba46caabc70e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:37.879735 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-c29d1d37-a7ed-4e53-a92c-ba46caabc70e None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:37.881453 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c29d1d37-a7ed-4e53-a92c-ba46caabc70e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:37.899397 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512d1ea9-4b89-4126-bacf-4e1ef94aa7c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:37.900370 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512d1ea9-4b89-4126-bacf-4e1ef94aa7c5 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:37.900653 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512d1ea9-4b89-4126-bacf-4e1ef94aa7c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:37.900924 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512d1ea9-4b89-4126-bacf-4e1ef94aa7c5 None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:37.901530 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512d1ea9-4b89-4126-bacf-4e1ef94aa7c5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:37.903443 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512d1ea9-4b89-4126-bacf-4e1ef94aa7c5 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:37.909893 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 392/780] 10.222.0.26 () {56 vars in 1087 bytes} [Tue Sep 27 04:07:37 2022] GET /identity/v3/services => generated 772 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:07:37.910687 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512d1ea9-4b89-4126-bacf-4e1ef94aa7c5 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:37.925849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:38.002968 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 389/781] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:37 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 04:07:38.074975 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512d1ea9-4b89-4126-bacf-4e1ef94aa7c5 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:38.110294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bef4f27-242f-48c7-8126-066c46872d7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:38.110817 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bef4f27-242f-48c7-8126-066c46872d7e None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:38.111095 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bef4f27-242f-48c7-8126-066c46872d7e None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:38.111363 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bef4f27-242f-48c7-8126-066c46872d7e None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:38.111971 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bef4f27-242f-48c7-8126-066c46872d7e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:38.113330 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bef4f27-242f-48c7-8126-066c46872d7e None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:38.119633 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 393/782] 10.222.0.26 () {56 vars in 1087 bytes} [Tue Sep 27 04:07:37 2022] GET /identity/v3/services => generated 772 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:38.351924 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29d1d37-a7ed-4e53-a92c-ba46caabc70e None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:38.456555 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:38.462239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29d1d37-a7ed-4e53-a92c-ba46caabc70e None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:38.502013 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4567a69-b535-4bcf-a039-674436c523d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:38.502736 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4567a69-b535-4bcf-a039-674436c523d7 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:38.503015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4567a69-b535-4bcf-a039-674436c523d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:38.503286 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4567a69-b535-4bcf-a039-674436c523d7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:38.515857 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4567a69-b535-4bcf-a039-674436c523d7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:38.520827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-d4567a69-b535-4bcf-a039-674436c523d7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:07:38.520827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:38.520827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:38.520827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:38.520827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:38.520827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:38.520827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:38.520827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:38.520827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:38.520827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:38.520827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:38.520827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:38.520827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:38.520827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:38.520827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:38.520827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:38.521827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:38.521827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:38.521827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:38.521827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:38.521827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:38.521827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:38.521827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:38.521827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:38.521827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:38.521827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:38.521827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:38.521827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:38.521827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:38.521827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:38.521827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:38.521827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:38.522672 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:38.522672 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:38.522672 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:38.522672 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:38.522672 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:38.522672 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:38.522672 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:38.522672 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:38.522672 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:38.522672 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:38.522672 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:07:38.522672 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:38.522672 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 390/783] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:07:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:38.528531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bef4f27-242f-48c7-8126-066c46872d7e None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:38.610351 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9848d4-0683-4a2e-9a9f-05b06ed77bba None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:38.610675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9848d4-0683-4a2e-9a9f-05b06ed77bba None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:38.610895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9848d4-0683-4a2e-9a9f-05b06ed77bba None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:38.614032 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 391/784] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:07:38.618172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0d0325-d097-4aca-b198-c03c8f3b70bc None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:38.618452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0d0325-d097-4aca-b198-c03c8f3b70bc None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:38.618727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0d0325-d097-4aca-b198-c03c8f3b70bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:38.621635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:38.646398 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-fc0d0325-d097-4aca-b198-c03c8f3b70bc None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:38.647918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-fc0d0325-d097-4aca-b198-c03c8f3b70bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:38.681877 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bef4f27-242f-48c7-8126-066c46872d7e None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:38.712140 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f17527d-2039-4c31-9b38-3629efb89ba7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:38.712752 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f17527d-2039-4c31-9b38-3629efb89ba7 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:38.713038 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f17527d-2039-4c31-9b38-3629efb89ba7 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:38.713314 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f17527d-2039-4c31-9b38-3629efb89ba7 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:38.714027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f17527d-2039-4c31-9b38-3629efb89ba7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:38.714330 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f17527d-2039-4c31-9b38-3629efb89ba7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:38.715782 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f17527d-2039-4c31-9b38-3629efb89ba7 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:38.725228 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 394/785] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:07:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:38.735051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f17527d-2039-4c31-9b38-3629efb89ba7 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:38.843114 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f17527d-2039-4c31-9b38-3629efb89ba7 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:38.847391 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 392/786] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:38 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:07:38.879359 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:38.887867 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ea4a57-9378-4201-9587-274391cf03eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:38.889051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ea4a57-9378-4201-9587-274391cf03eb None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:38.889568 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ea4a57-9378-4201-9587-274391cf03eb None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:38.890148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ea4a57-9378-4201-9587-274391cf03eb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:38.906747 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ea4a57-9378-4201-9587-274391cf03eb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:38.908666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ea4a57-9378-4201-9587-274391cf03eb None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:38.931486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-b7ea4a57-9378-4201-9587-274391cf03eb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:38.931486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:38.931486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:38.931486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:38.931486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:38.931486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:38.931486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:38.931486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:38.931486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:38.931486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:38.931486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:38.931486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:38.931486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:38.931486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:38.931486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:38.931486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:38.932510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:38.932510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:38.932510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:38.932510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:38.932510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:38.932510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:38.932510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:38.932510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:38.932510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:38.932510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:38.932510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:38.932510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:38.932510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:38.932510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:38.932510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:38.932510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:38.932510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:38.933396 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:38.933396 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:38.933396 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:38.939329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 395/787] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:07:38.945599 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0d0325-d097-4aca-b198-c03c8f3b70bc None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:38.998324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646772d5-918b-4fa7-ba24-24d7dd314a88 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:38.998969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646772d5-918b-4fa7-ba24-24d7dd314a88 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:38.999705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646772d5-918b-4fa7-ba24-24d7dd314a88 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:39.002414 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 396/788] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:38 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:07:39.007988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981b423d-ccc1-4391-b169-643bd04ede97 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:39.008260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981b423d-ccc1-4391-b169-643bd04ede97 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:39.008531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981b423d-ccc1-4391-b169-643bd04ede97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:39.045229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-981b423d-ccc1-4391-b169-643bd04ede97 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:39.047336 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-981b423d-ccc1-4391-b169-643bd04ede97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:39.098106 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:39.116435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0d0325-d097-4aca-b198-c03c8f3b70bc None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:39.175283 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0b582c-3598-4e0f-8ce4-7b3fada0a5b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:39.175901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0b582c-3598-4e0f-8ce4-7b3fada0a5b0 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:39.176175 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0b582c-3598-4e0f-8ce4-7b3fada0a5b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:39.176445 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0b582c-3598-4e0f-8ce4-7b3fada0a5b0 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:39.177130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0b582c-3598-4e0f-8ce4-7b3fada0a5b0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:39.177428 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0b582c-3598-4e0f-8ce4-7b3fada0a5b0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:39.178814 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0b582c-3598-4e0f-8ce4-7b3fada0a5b0 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:39.183932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 397/789] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:39 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 04:07:39.198510 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 393/790] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:39.203938 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981b423d-ccc1-4391-b169-643bd04ede97 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:39.207627 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0b582c-3598-4e0f-8ce4-7b3fada0a5b0 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:39.376685 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981b423d-ccc1-4391-b169-643bd04ede97 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:39.410108 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:39.413816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0b582c-3598-4e0f-8ce4-7b3fada0a5b0 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:39.456990 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19dad948-58b2-41c9-a6d5-4ffa0e1012c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:39.457544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19dad948-58b2-41c9-a6d5-4ffa0e1012c2 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:39.457846 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19dad948-58b2-41c9-a6d5-4ffa0e1012c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:39.458147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19dad948-58b2-41c9-a6d5-4ffa0e1012c2 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:39.469069 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19dad948-58b2-41c9-a6d5-4ffa0e1012c2 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:39.470523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19dad948-58b2-41c9-a6d5-4ffa0e1012c2 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:39.490347 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-19dad948-58b2-41c9-a6d5-4ffa0e1012c2 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 27 04:07:39.490347 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:39.490347 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:39.490347 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:39.490347 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:39.490347 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:39.490347 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:39.490347 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:39.490347 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:39.490347 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:39.490347 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:39.490347 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:39.490347 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:39.490347 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:39.490347 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:39.490347 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:39.491313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:39.491313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:39.491313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:39.491313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:39.491313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:39.491313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:39.491313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:39.491313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:39.491313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:39.491313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:39.491313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:39.491313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:39.491313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:39.491313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:39.491313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:39.491313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:39.491313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:39.491313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:39.492293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:39.492293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:39.492293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:39.492293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:39.492293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:39.492293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:39.492293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:39.492293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:39.492293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 27 04:07:39.492293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:39.492293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 398/791] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:07:39 2022] GET /identity/v3/users/swift => generated 83 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:39.494168 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f464fac-ae07-4b95-84f1-e77e4346e78e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:39.494799 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f464fac-ae07-4b95-84f1-e77e4346e78e None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:39.495072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f464fac-ae07-4b95-84f1-e77e4346e78e None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:39.495340 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f464fac-ae07-4b95-84f1-e77e4346e78e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:39.498626 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19dad948-58b2-41c9-a6d5-4ffa0e1012c2 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:39.511381 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f464fac-ae07-4b95-84f1-e77e4346e78e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:39.518264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-0f464fac-ae07-4b95-84f1-e77e4346e78e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:39.518264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:39.518264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:39.518264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:39.518264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:39.518264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:39.518264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:39.518264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:39.518264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:39.518264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:39.518264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:39.518264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:39.518264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:39.518264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:39.518264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:39.518264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:39.519181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:39.519181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:39.519181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:39.519181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:39.519181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:39.519181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:39.519181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:39.519181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:39.519181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:39.519181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:39.519181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:39.519181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:39.519181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:39.519181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:39.519181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:39.519181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:39.520016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:39.520016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:39.520016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:39.520016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:39.520016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:39.520016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:39.520016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:39.520016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:39.520016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:39.520016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:39.520016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:39.520016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:39.520686 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 394/792] 10.222.0.26 () {56 vars in 1102 bytes} [Tue Sep 27 04:07:39 2022] GET /identity/v3/roles/service => generated 85 bytes in 313 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:07:39.521210 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f464fac-ae07-4b95-84f1-e77e4346e78e None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:39.546105 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:39.618498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19dad948-58b2-41c9-a6d5-4ffa0e1012c2 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:39.661942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f29642d-dffa-444a-8341-1f280d88fa52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:39.662552 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f29642d-dffa-444a-8341-1f280d88fa52 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:39.662833 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f29642d-dffa-444a-8341-1f280d88fa52 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:39.663108 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f29642d-dffa-444a-8341-1f280d88fa52 None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:39.663837 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f29642d-dffa-444a-8341-1f280d88fa52 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:39.664143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f29642d-dffa-444a-8341-1f280d88fa52 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:39.664433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f464fac-ae07-4b95-84f1-e77e4346e78e None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:39.665569 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f29642d-dffa-444a-8341-1f280d88fa52 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:39.694962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 399/793] 10.222.0.26 () {56 vars in 1135 bytes} [Tue Sep 27 04:07:39 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:39.694962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f29642d-dffa-444a-8341-1f280d88fa52 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:39.705629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36123d39-f2ed-4121-bc54-f1a5098ccba1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:39.706277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36123d39-f2ed-4121-bc54-f1a5098ccba1 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:39.706554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36123d39-f2ed-4121-bc54-f1a5098ccba1 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:39.706825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36123d39-f2ed-4121-bc54-f1a5098ccba1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:39.720264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36123d39-f2ed-4121-bc54-f1a5098ccba1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:39.728152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-36123d39-f2ed-4121-bc54-f1a5098ccba1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:07:39.728152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:39.728152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:39.728152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:39.728152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:39.728152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:39.728152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:39.728152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:39.728152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:39.728152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:39.728152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:39.728152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:39.728152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:39.728152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:39.728152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:39.728152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:39.729078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:39.729078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:39.729078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:39.729078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:39.729078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:39.729078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:39.729078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:39.729078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:39.729078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:39.729078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:39.729078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:39.729078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:39.729078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:39.729078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:39.729078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:39.729078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:39.729953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:39.729953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:39.729953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:39.729953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:39.729953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:39.729953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:39.729953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:39.729953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:39.729953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:39.729953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:39.729953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:07:39.729953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:39.733822 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 395/794] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:07:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:39.734079 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:39.735683 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36123d39-f2ed-4121-bc54-f1a5098ccba1 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:39.812090 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f29642d-dffa-444a-8341-1f280d88fa52 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:39.814166 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:39.855538 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36123d39-f2ed-4121-bc54-f1a5098ccba1 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:39.855754 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98d4472-33ea-4bc2-9202-e0ff6d84ac32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:39.856652 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98d4472-33ea-4bc2-9202-e0ff6d84ac32 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:39.856999 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98d4472-33ea-4bc2-9202-e0ff6d84ac32 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:39.857335 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98d4472-33ea-4bc2-9202-e0ff6d84ac32 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:39.858126 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98d4472-33ea-4bc2-9202-e0ff6d84ac32 None admin] RBAC: Adding query filter params (name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:39.858460 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98d4472-33ea-4bc2-9202-e0ff6d84ac32 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:39.859922 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98d4472-33ea-4bc2-9202-e0ff6d84ac32 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:39.872668 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 400/795] 10.222.0.26 () {56 vars in 1103 bytes} [Tue Sep 27 04:07:39 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:39.873574 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98d4472-33ea-4bc2-9202-e0ff6d84ac32 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:39.899299 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221e56fb-3885-4ead-bd1a-bc482d36ab68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:39.900527 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221e56fb-3885-4ead-bd1a-bc482d36ab68 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:39.901244 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221e56fb-3885-4ead-bd1a-bc482d36ab68 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:39.901716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221e56fb-3885-4ead-bd1a-bc482d36ab68 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:39.915515 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221e56fb-3885-4ead-bd1a-bc482d36ab68 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:39.917532 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221e56fb-3885-4ead-bd1a-bc482d36ab68 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:39.934583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-221e56fb-3885-4ead-bd1a-bc482d36ab68 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:39.934583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:39.934583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:39.934583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:39.934583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:39.934583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:39.934583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:39.934583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:39.934583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:39.934583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:39.934583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:39.934583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:39.934583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:39.934583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:39.934583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:39.934583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:39.935540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:39.935540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:39.935540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:39.935540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:39.935540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:39.935540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:39.935540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:39.935540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:39.935540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:39.935540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:39.935540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:39.935540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:39.935540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:39.935540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:39.935540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:39.935540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:39.935540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:39.936414 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:39.936414 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:39.936414 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:39.936414 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 396/796] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:39.936414 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221e56fb-3885-4ead-bd1a-bc482d36ab68 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:39.990099 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:39.998117 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98d4472-33ea-4bc2-9202-e0ff6d84ac32 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:40.034006 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54fec08-b55c-491f-a76a-18e210cdb016 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:40.034606 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54fec08-b55c-491f-a76a-18e210cdb016 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:40.034881 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54fec08-b55c-491f-a76a-18e210cdb016 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:40.035149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54fec08-b55c-491f-a76a-18e210cdb016 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:40.035805 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54fec08-b55c-491f-a76a-18e210cdb016 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:40.036121 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54fec08-b55c-491f-a76a-18e210cdb016 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:40.037463 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54fec08-b55c-491f-a76a-18e210cdb016 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:40.050600 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 401/797] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:07:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:40.050600 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54fec08-b55c-491f-a76a-18e210cdb016 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:40.072889 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221e56fb-3885-4ead-bd1a-bc482d36ab68 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:40.093145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:40.108848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3009c1a0-b6e2-4bac-968d-bff0c6248f6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:40.109667 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3009c1a0-b6e2-4bac-968d-bff0c6248f6e None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:40.110062 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3009c1a0-b6e2-4bac-968d-bff0c6248f6e None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:40.110714 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3009c1a0-b6e2-4bac-968d-bff0c6248f6e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:40.123177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3009c1a0-b6e2-4bac-968d-bff0c6248f6e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:40.124762 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3009c1a0-b6e2-4bac-968d-bff0c6248f6e None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:40.140362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-3009c1a0-b6e2-4bac-968d-bff0c6248f6e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:40.140362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:40.140362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:40.140362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:40.140362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:40.140362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:40.140362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:40.140362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:40.140362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:40.140362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:40.140362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:40.140362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:40.140362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:40.140362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:40.140362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:40.140362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:40.141400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:40.141400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:40.141400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:40.141400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:40.141400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:40.141400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:40.141400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:40.141400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:40.141400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:40.141400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:40.141400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:40.141400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:40.141400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:40.141400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:40.141400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:40.141400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:40.141400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:40.142354 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:40.142354 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:40.142354 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:40.146882 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 397/798] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:40.147868 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3009c1a0-b6e2-4bac-968d-bff0c6248f6e None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:40.173294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54fec08-b55c-491f-a76a-18e210cdb016 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:40.212593 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a384c8-bc38-4e86-949f-6c2ba6ea44ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:40.213686 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a384c8-bc38-4e86-949f-6c2ba6ea44ba None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:40.214138 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a384c8-bc38-4e86-949f-6c2ba6ea44ba None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:40.214555 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a384c8-bc38-4e86-949f-6c2ba6ea44ba None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:40.215364 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a384c8-bc38-4e86-949f-6c2ba6ea44ba None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:40.215821 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a384c8-bc38-4e86-949f-6c2ba6ea44ba None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:40.217304 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a384c8-bc38-4e86-949f-6c2ba6ea44ba None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:40.232822 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 402/799] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:40.233958 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a384c8-bc38-4e86-949f-6c2ba6ea44ba None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:40.255560 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:40.258469 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3009c1a0-b6e2-4bac-968d-bff0c6248f6e None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:40.295072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1056ff0b-1c73-4722-9907-392c47289691 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:40.295867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1056ff0b-1c73-4722-9907-392c47289691 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:40.296263 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1056ff0b-1c73-4722-9907-392c47289691 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:40.296621 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1056ff0b-1c73-4722-9907-392c47289691 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:40.308719 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1056ff0b-1c73-4722-9907-392c47289691 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:40.310334 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1056ff0b-1c73-4722-9907-392c47289691 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:40.326859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-1056ff0b-1c73-4722-9907-392c47289691 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:40.326859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:40.326859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:40.326859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:40.326859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:40.326859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:40.326859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:40.326859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:40.326859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:40.326859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:40.326859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:40.326859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:40.326859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:40.326859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:40.326859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:40.326859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:40.328084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:40.328084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:40.328084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:40.328084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:40.328084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:40.328084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:40.328084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:40.328084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:40.328084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:40.328084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:40.328084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:40.328084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:40.328084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:40.328084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:40.328084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:40.328084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:40.328084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:40.329179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:40.329179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:40.329179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:40.334838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 398/800] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:40.334838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1056ff0b-1c73-4722-9907-392c47289691 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:40.358683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a384c8-bc38-4e86-949f-6c2ba6ea44ba None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:40.359384 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:40.405480 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8785ac5-7c9f-4da6-974b-1bd61cc915d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:40.406286 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8785ac5-7c9f-4da6-974b-1bd61cc915d6 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:40.407270 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8785ac5-7c9f-4da6-974b-1bd61cc915d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:40.407523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8785ac5-7c9f-4da6-974b-1bd61cc915d6 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:40.408584 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8785ac5-7c9f-4da6-974b-1bd61cc915d6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:40.409003 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8785ac5-7c9f-4da6-974b-1bd61cc915d6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:40.410522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8785ac5-7c9f-4da6-974b-1bd61cc915d6 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:40.428024 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 403/801] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:40.429128 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8785ac5-7c9f-4da6-974b-1bd61cc915d6 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:40.460679 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1056ff0b-1c73-4722-9907-392c47289691 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:40.497978 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87316834-3b42-41b5-9b15-966fa7c2b32b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:40.498776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87316834-3b42-41b5-9b15-966fa7c2b32b None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:40.499142 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87316834-3b42-41b5-9b15-966fa7c2b32b None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:40.499591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87316834-3b42-41b5-9b15-966fa7c2b32b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:40.507608 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87316834-3b42-41b5-9b15-966fa7c2b32b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:40.508970 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87316834-3b42-41b5-9b15-966fa7c2b32b None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:40.520249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-87316834-3b42-41b5-9b15-966fa7c2b32b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:40.520249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:40.520249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:40.520249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:40.520249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:40.520249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:40.520249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:40.520249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:40.520249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:40.520249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:40.520249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:40.520249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:40.520249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:40.520249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:40.520249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:40.521346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:40.521346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:40.521346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:40.521346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:40.521346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:40.521346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:40.521346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:40.521346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:40.521346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:40.521346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:40.521346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:40.521346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:40.521346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:40.521346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:40.521346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:40.521346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:40.521346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:40.522324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:40.522324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:40.522324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:40.522324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:40.522324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:40.522324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:40.522324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:40.522324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:40.522324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 399/802] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:40.522816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87316834-3b42-41b5-9b15-966fa7c2b32b None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:40.527580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:40.556869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8785ac5-7c9f-4da6-974b-1bd61cc915d6 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:40.599169 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651a0634-081c-4fa5-a55c-f9589f21f470 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:40.600072 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651a0634-081c-4fa5-a55c-f9589f21f470 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:40.600498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651a0634-081c-4fa5-a55c-f9589f21f470 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:40.600921 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651a0634-081c-4fa5-a55c-f9589f21f470 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:40.601787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651a0634-081c-4fa5-a55c-f9589f21f470 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:40.602858 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651a0634-081c-4fa5-a55c-f9589f21f470 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:40.604757 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651a0634-081c-4fa5-a55c-f9589f21f470 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:40.612218 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:40.621692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 404/803] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:40.624287 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ecc786-df10-457d-b75a-a86f8aab4a4c None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:40.624699 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ecc786-df10-457d-b75a-a86f8aab4a4c None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:40.625125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ecc786-df10-457d-b75a-a86f8aab4a4c None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:40.626603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 405/804] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:40.627471 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ecc786-df10-457d-b75a-a86f8aab4a4c None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:40.633697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87316834-3b42-41b5-9b15-966fa7c2b32b None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:40.668940 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e0a8eb-6a82-4b30-922f-d99a71cdf631 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:40.669924 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e0a8eb-6a82-4b30-922f-d99a71cdf631 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:40.670204 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e0a8eb-6a82-4b30-922f-d99a71cdf631 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:40.670475 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e0a8eb-6a82-4b30-922f-d99a71cdf631 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:40.678691 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e0a8eb-6a82-4b30-922f-d99a71cdf631 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:40.680558 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e0a8eb-6a82-4b30-922f-d99a71cdf631 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:40.691754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-b4e0a8eb-6a82-4b30-922f-d99a71cdf631 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 27 04:07:40.691754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:40.691754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:40.691754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:40.691754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:40.691754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:40.691754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:40.691754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:40.691754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:40.691754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:40.691754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:40.691754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:40.691754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:40.691754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:40.691754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:40.691754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:40.692734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:40.692734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:40.692734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:40.692734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:40.692734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:40.692734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:40.692734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:40.692734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:40.692734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:40.692734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:40.692734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:40.692734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:40.692734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:40.692734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:40.692734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:40.692734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:40.692734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:40.692734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:40.693698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:40.693698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:40.693698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:40.693698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:40.693698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:40.693698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:40.693698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:40.693698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:40.693698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 27 04:07:40.693698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:40.693698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 400/805] 10.222.0.26 () {56 vars in 1117 bytes} [Tue Sep 27 04:07:40 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:40.693698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e0a8eb-6a82-4b30-922f-d99a71cdf631 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:40.741973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ecc786-df10-457d-b75a-a86f8aab4a4c None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:40.775608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107437f8-0cb9-45e8-a318-f373861e0a35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:40.776139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107437f8-0cb9-45e8-a318-f373861e0a35 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:40.776420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107437f8-0cb9-45e8-a318-f373861e0a35 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:40.776690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107437f8-0cb9-45e8-a318-f373861e0a35 None admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:40.777370 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107437f8-0cb9-45e8-a318-f373861e0a35 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:40.777670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107437f8-0cb9-45e8-a318-f373861e0a35 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:40.779117 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107437f8-0cb9-45e8-a318-f373861e0a35 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:40.781536 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:40.792006 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 406/806] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:40.793403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f657f4-6ff3-4213-8707-fd8519e5eab7 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:40.793682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f657f4-6ff3-4213-8707-fd8519e5eab7 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:40.793983 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f657f4-6ff3-4213-8707-fd8519e5eab7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:40.799001 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e0a8eb-6a82-4b30-922f-d99a71cdf631 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:40.811802 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-20f657f4-6ff3-4213-8707-fd8519e5eab7 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:40.812986 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-20f657f4-6ff3-4213-8707-fd8519e5eab7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:40.837455 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f499bfb7-8a61-43b6-9067-0931712c15cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:40.838111 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f499bfb7-8a61-43b6-9067-0931712c15cb None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:40.838386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f499bfb7-8a61-43b6-9067-0931712c15cb None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:40.838657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f499bfb7-8a61-43b6-9067-0931712c15cb None admin] PATH_INFO: `/v3/users/nova` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:40.846760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f499bfb7-8a61-43b6-9067-0931712c15cb None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:40.848300 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f499bfb7-8a61-43b6-9067-0931712c15cb None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:40.861293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-f499bfb7-8a61-43b6-9067-0931712c15cb None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 27 04:07:40.861293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:40.861293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:40.861293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:40.861293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:40.861293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:40.861293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:40.861293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:40.861293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:40.861293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:40.861293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:40.861293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:40.861293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:40.861293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:40.861293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:40.861293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:40.862314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:40.862314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:40.862314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:40.862314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:40.862314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:40.862314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:40.862314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:40.862314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:40.862314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:40.862314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:40.862314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:40.862314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:40.862314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:40.862314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:40.862314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:40.862314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:40.862314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:40.862314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:40.863253 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:40.863253 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:40.863253 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:40.863253 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:40.863253 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:40.863253 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:40.863253 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:40.863253 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:40.863253 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 27 04:07:40.863253 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:40.863253 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 401/807] 10.222.0.26 () {56 vars in 1093 bytes} [Tue Sep 27 04:07:40 2022] GET /identity/v3/users/nova => generated 82 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:40.864413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0024c6e-6382-4ddc-9f3f-54d5f6fbd70c None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:40.864661 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0024c6e-6382-4ddc-9f3f-54d5f6fbd70c None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:40.864932 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0024c6e-6382-4ddc-9f3f-54d5f6fbd70c None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:40.866165 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 402/808] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:40 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:40.866846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0024c6e-6382-4ddc-9f3f-54d5f6fbd70c None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:40.871363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:40.887748 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 407/809] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:40 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:07:40.888560 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f657f4-6ff3-4213-8707-fd8519e5eab7 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:40.971299 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0024c6e-6382-4ddc-9f3f-54d5f6fbd70c None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:41.005870 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ab81b0-c2f4-467f-9fc2-1ad997778f96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:41.006468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ab81b0-c2f4-467f-9fc2-1ad997778f96 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:41.006740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ab81b0-c2f4-467f-9fc2-1ad997778f96 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:41.007011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ab81b0-c2f4-467f-9fc2-1ad997778f96 None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:41.007720 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ab81b0-c2f4-467f-9fc2-1ad997778f96 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:41.008018 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ab81b0-c2f4-467f-9fc2-1ad997778f96 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:41.009429 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ab81b0-c2f4-467f-9fc2-1ad997778f96 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:41.010200 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f657f4-6ff3-4213-8707-fd8519e5eab7 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:41.032113 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 403/810] 10.222.0.26 () {56 vars in 1149 bytes} [Tue Sep 27 04:07:40 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:41.032113 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ab81b0-c2f4-467f-9fc2-1ad997778f96 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:41.042822 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:41.053386 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29341ebb-dd1b-4e87-9ae1-cfce52e86540 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:41.054245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29341ebb-dd1b-4e87-9ae1-cfce52e86540 None admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:41.054603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29341ebb-dd1b-4e87-9ae1-cfce52e86540 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:41.054817 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29341ebb-dd1b-4e87-9ae1-cfce52e86540 None admin] PATH_INFO: `/v3/projects/c830a3bfece6456486f4b094076b5364/users/181dd5dc720c4a97860852062b7b19c3/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:41.056750 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29341ebb-dd1b-4e87-9ae1-cfce52e86540 None admin] RBAC: Authorizing `identity:create_grant(project_id=c830a3bfece6456486f4b094076b5364, user_id=181dd5dc720c4a97860852062b7b19c3, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:41.058242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29341ebb-dd1b-4e87-9ae1-cfce52e86540 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:41.071537 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 408/811] 10.222.0.26 () {58 vars in 1439 bytes} [Tue Sep 27 04:07:40 2022] PUT /identity/v3/projects/c830a3bfece6456486f4b094076b5364/users/181dd5dc720c4a97860852062b7b19c3/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:07:41.075016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5637726-ad39-47a4-be1c-6b4dd89fba95 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:41.075300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5637726-ad39-47a4-be1c-6b4dd89fba95 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:41.075566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5637726-ad39-47a4-be1c-6b4dd89fba95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:41.100854 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:41.109345 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-b5637726-ad39-47a4-be1c-6b4dd89fba95 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:41.110593 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-b5637726-ad39-47a4-be1c-6b4dd89fba95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:41.162020 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ab81b0-c2f4-467f-9fc2-1ad997778f96 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:41.190353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 409/812] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:41 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:07:41.190353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5637726-ad39-47a4-be1c-6b4dd89fba95 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:41.207934 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56701392-d71b-4ad2-a4e3-590ee4fb09c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:41.209690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56701392-d71b-4ad2-a4e3-590ee4fb09c4 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:41.210134 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56701392-d71b-4ad2-a4e3-590ee4fb09c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:41.211017 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56701392-d71b-4ad2-a4e3-590ee4fb09c4 None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:41.212181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56701392-d71b-4ad2-a4e3-590ee4fb09c4 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:41.212681 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56701392-d71b-4ad2-a4e3-590ee4fb09c4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:41.214987 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56701392-d71b-4ad2-a4e3-590ee4fb09c4 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:41.245051 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 404/813] 10.222.0.26 () {56 vars in 1133 bytes} [Tue Sep 27 04:07:41 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:41.245051 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56701392-d71b-4ad2-a4e3-590ee4fb09c4 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:41.316216 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5637726-ad39-47a4-be1c-6b4dd89fba95 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:41.334255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:41.353351 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7072608-ccef-4b66-b45d-b9b51b95c13e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:41.354053 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7072608-ccef-4b66-b45d-b9b51b95c13e None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:41.354330 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7072608-ccef-4b66-b45d-b9b51b95c13e None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:41.354598 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7072608-ccef-4b66-b45d-b9b51b95c13e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:41.365101 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7072608-ccef-4b66-b45d-b9b51b95c13e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:41.366845 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7072608-ccef-4b66-b45d-b9b51b95c13e None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:41.382280 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56701392-d71b-4ad2-a4e3-590ee4fb09c4 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:41.382630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-d7072608-ccef-4b66-b45d-b9b51b95c13e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:41.382630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:41.382630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:41.382630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:41.382630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:41.382630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:41.382630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:41.382630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:41.382630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:41.382630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:41.382630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:41.382630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:41.382630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:41.382630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:41.382630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:41.382630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:41.383567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:41.383567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:41.383567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:41.383567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:41.383567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:41.383567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:41.383567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:41.383567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:41.383567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:41.383567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:41.383567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:41.383567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:41.383567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:41.383567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:41.383567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:41.383567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:41.383567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:41.384932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:41.384932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:41.384932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:41.384932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 410/814] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:41.384932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7072608-ccef-4b66-b45d-b9b51b95c13e None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:41.415806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:41.436924 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6351e3-af10-4c5a-8a2d-309d546d2fa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:41.438437 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6351e3-af10-4c5a-8a2d-309d546d2fa9 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:41.438964 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6351e3-af10-4c5a-8a2d-309d546d2fa9 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:41.439441 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6351e3-af10-4c5a-8a2d-309d546d2fa9 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:41.440671 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd6351e3-af10-4c5a-8a2d-309d546d2fa9 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:41.443533 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd6351e3-af10-4c5a-8a2d-309d546d2fa9 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:41.455915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-bd6351e3-af10-4c5a-8a2d-309d546d2fa9 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 27 04:07:41.455915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:41.455915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:07:41.455915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:07:41.455915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:07:41.455915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:07:41.455915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:07:41.455915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:41.455915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 27 04:07:41.455915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:41.455915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:07:41.455915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:41.455915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:41.455915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:41.455915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:41.455915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:41.455915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:41.456950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:41.456950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:41.456950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:41.456950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:41.456950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:41.456950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:41.456950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:07:41.456950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:07:41.456950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:07:41.456950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:07:41.456950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:41.456950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:41.456950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:41.456950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:41.456950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:41.456950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:41.457825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:41.457825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:41.457825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:41.457825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:41.457825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:41.457825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:41.457825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:41.457825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:41.457825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:41.457825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:41.457825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:07:41.457825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:41.457825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 27 04:07:41.457825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:41.460434 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 405/815] 10.222.0.26 () {56 vars in 1129 bytes} [Tue Sep 27 04:07:41 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:41.460648 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6351e3-af10-4c5a-8a2d-309d546d2fa9 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:41.510616 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7072608-ccef-4b66-b45d-b9b51b95c13e None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:41.552704 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c921fd17-0714-4551-a1ba-803fdc481c83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:41.553642 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c921fd17-0714-4551-a1ba-803fdc481c83 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:41.554168 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c921fd17-0714-4551-a1ba-803fdc481c83 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:41.554602 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c921fd17-0714-4551-a1ba-803fdc481c83 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:41.568588 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6351e3-af10-4c5a-8a2d-309d546d2fa9 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:41.592119 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:41.601449 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12efe3b8-edec-4204-8f64-7399c9b276e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:41.602084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12efe3b8-edec-4204-8f64-7399c9b276e9 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:41.602366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12efe3b8-edec-4204-8f64-7399c9b276e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:41.602645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12efe3b8-edec-4204-8f64-7399c9b276e9 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:41.603310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12efe3b8-edec-4204-8f64-7399c9b276e9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:41.603609 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12efe3b8-edec-4204-8f64-7399c9b276e9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:41.604978 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12efe3b8-edec-4204-8f64-7399c9b276e9 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:41.618870 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 406/816] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:41.619903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12efe3b8-edec-4204-8f64-7399c9b276e9 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:41.638533 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c921fd17-0714-4551-a1ba-803fdc481c83 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:41.641325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c921fd17-0714-4551-a1ba-803fdc481c83 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:41.668836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:41.723111 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12efe3b8-edec-4204-8f64-7399c9b276e9 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:41.750324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 411/817] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:07:41 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 04:07:41.755808 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c921fd17-0714-4551-a1ba-803fdc481c83 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:41.763985 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89939ebf-852f-48b9-8b9c-20146a4b118d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:41.764583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89939ebf-852f-48b9-8b9c-20146a4b118d None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:41.764855 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89939ebf-852f-48b9-8b9c-20146a4b118d None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:41.765131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89939ebf-852f-48b9-8b9c-20146a4b118d None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:41.765776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89939ebf-852f-48b9-8b9c-20146a4b118d None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:41.766083 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89939ebf-852f-48b9-8b9c-20146a4b118d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:41.767433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89939ebf-852f-48b9-8b9c-20146a4b118d None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:41.772190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 407/818] 10.222.0.26 () {56 vars in 1124 bytes} [Tue Sep 27 04:07:41 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:41.772950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89939ebf-852f-48b9-8b9c-20146a4b118d None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:41.829280 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:41.865438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c921fd17-0714-4551-a1ba-803fdc481c83 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:41.873876 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89939ebf-852f-48b9-8b9c-20146a4b118d None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:41.883954 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:41.901317 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db1d5fe-1545-4acc-9318-7581ad80e5ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:41.901963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db1d5fe-1545-4acc-9318-7581ad80e5ad None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:41.902241 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db1d5fe-1545-4acc-9318-7581ad80e5ad None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:41.902509 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db1d5fe-1545-4acc-9318-7581ad80e5ad None admin] PATH_INFO: `/v3/services/network` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:41.903083 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db1d5fe-1545-4acc-9318-7581ad80e5ad None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:41.904417 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db1d5fe-1545-4acc-9318-7581ad80e5ad None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:41.910808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e514785-84e9-4b1d-a885-e69c3b92cda7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:41.911642 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e514785-84e9-4b1d-a885-e69c3b92cda7 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:41.912055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e514785-84e9-4b1d-a885-e69c3b92cda7 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:41.912472 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e514785-84e9-4b1d-a885-e69c3b92cda7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:41.912953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-0db1d5fe-1545-4acc-9318-7581ad80e5ad None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 27 04:07:41.912953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:41.912953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:07:41.912953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:07:41.912953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:07:41.912953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:07:41.912953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:07:41.912953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:41.912953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 27 04:07:41.912953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:41.912953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:07:41.912953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:41.912953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:41.912953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:41.912953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:41.912953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:41.912953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:41.914100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:41.914100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:41.914100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:41.914100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:41.914100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:41.914100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:41.914100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:07:41.914100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:07:41.914100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:07:41.914100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:07:41.914100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:41.914100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:41.914100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:41.914100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:41.914100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:41.914100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:41.915016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:41.915016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:41.915016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:41.915016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:41.915016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:41.915016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:41.915016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:41.915016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:41.915016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:41.915016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:41.915016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:07:41.915016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:41.915016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 27 04:07:41.915016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:41.915016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 412/819] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:41 2022] GET /identity/v3/services/network => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:41.915801 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db1d5fe-1545-4acc-9318-7581ad80e5ad None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:41.926422 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e514785-84e9-4b1d-a885-e69c3b92cda7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:41.928427 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e514785-84e9-4b1d-a885-e69c3b92cda7 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:41.939470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-7e514785-84e9-4b1d-a885-e69c3b92cda7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:41.939470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:41.939470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:41.939470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:41.939470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:41.939470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:41.939470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:41.939470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:41.939470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:41.939470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:41.939470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:41.939470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:41.939470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:41.939470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:41.939470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:41.940515 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:41.940515 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:41.940515 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:41.940515 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:41.940515 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:41.940515 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:41.940515 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:41.940515 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:41.940515 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:41.940515 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:41.940515 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:41.940515 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:41.940515 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:41.940515 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:41.940515 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:41.940515 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:41.940515 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:41.941610 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:41.941610 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:41.941610 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:41.941610 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:41.941610 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:41.941610 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:41.941610 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:41.941610 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:41.945736 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 408/820] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:41.946551 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e514785-84e9-4b1d-a885-e69c3b92cda7 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:42.029195 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db1d5fe-1545-4acc-9318-7581ad80e5ad None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:42.050845 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e514785-84e9-4b1d-a885-e69c3b92cda7 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:42.063374 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:42.068863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee208d9-a274-4eff-83b5-3668383434ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:42.069518 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee208d9-a274-4eff-83b5-3668383434ac None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:42.069820 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee208d9-a274-4eff-83b5-3668383434ac None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:42.070093 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee208d9-a274-4eff-83b5-3668383434ac None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:42.081592 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee208d9-a274-4eff-83b5-3668383434ac None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:42.083257 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee208d9-a274-4eff-83b5-3668383434ac None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:42.086896 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4831e365-36cc-49c4-8e58-1d7251ac5162 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:42.087499 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4831e365-36cc-49c4-8e58-1d7251ac5162 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:42.087777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4831e365-36cc-49c4-8e58-1d7251ac5162 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:42.088050 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4831e365-36cc-49c4-8e58-1d7251ac5162 None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:42.088670 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4831e365-36cc-49c4-8e58-1d7251ac5162 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:42.088970 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4831e365-36cc-49c4-8e58-1d7251ac5162 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:42.090752 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4831e365-36cc-49c4-8e58-1d7251ac5162 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:42.101112 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 409/821] 10.222.0.26 () {56 vars in 1124 bytes} [Tue Sep 27 04:07:41 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:42.101112 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4831e365-36cc-49c4-8e58-1d7251ac5162 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:42.101357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-6ee208d9-a274-4eff-83b5-3668383434ac None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:42.101357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:42.101357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:42.101357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:42.101357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:42.101357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:42.101357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:42.101357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:42.101357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:42.101357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:42.101357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:42.101357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:42.101357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:42.101357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:42.101357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:42.101357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:42.102466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:42.102466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:42.102466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:42.102466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:42.102466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:42.102466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:42.102466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:42.102466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:42.102466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:42.102466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:42.102466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:42.102466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:42.102466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:42.102466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:42.102466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:42.102466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:42.102466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:42.103558 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:42.103558 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:42.103558 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:42.103558 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 413/822] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:42.103558 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee208d9-a274-4eff-83b5-3668383434ac None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:42.137517 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:42.211289 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4831e365-36cc-49c4-8e58-1d7251ac5162 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:42.212267 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee208d9-a274-4eff-83b5-3668383434ac None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:42.248029 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16274d54-1970-46e0-9fb3-5dde9de8a897 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:42.248250 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9db181f-02b6-4eec-bd21-f92415355d60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:42.248625 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16274d54-1970-46e0-9fb3-5dde9de8a897 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:42.248729 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9db181f-02b6-4eec-bd21-f92415355d60 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:42.248897 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16274d54-1970-46e0-9fb3-5dde9de8a897 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:42.248994 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9db181f-02b6-4eec-bd21-f92415355d60 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:42.249166 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16274d54-1970-46e0-9fb3-5dde9de8a897 None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:42.249262 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9db181f-02b6-4eec-bd21-f92415355d60 None admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:42.249808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16274d54-1970-46e0-9fb3-5dde9de8a897 None admin] RBAC: Adding query filter params (name=network) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:42.249905 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9db181f-02b6-4eec-bd21-f92415355d60 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:42.250106 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16274d54-1970-46e0-9fb3-5dde9de8a897 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:42.250213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9db181f-02b6-4eec-bd21-f92415355d60 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:42.251442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16274d54-1970-46e0-9fb3-5dde9de8a897 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:42.251665 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9db181f-02b6-4eec-bd21-f92415355d60 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:42.261119 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 410/823] 10.222.0.26 () {56 vars in 1112 bytes} [Tue Sep 27 04:07:42 2022] GET /identity/v3/services?name=network => generated 124 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:42.261119 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16274d54-1970-46e0-9fb3-5dde9de8a897 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:42.272432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 414/824] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:42.272432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9db181f-02b6-4eec-bd21-f92415355d60 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:42.365582 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:42.373089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16274d54-1970-46e0-9fb3-5dde9de8a897 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:42.418698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422b65a0-b911-4824-936b-c131e22e5779 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:42.418698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422b65a0-b911-4824-936b-c131e22e5779 None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:42.418698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422b65a0-b911-4824-936b-c131e22e5779 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:42.418698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422b65a0-b911-4824-936b-c131e22e5779 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:42.419121 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422b65a0-b911-4824-936b-c131e22e5779 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:42.421185 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422b65a0-b911-4824-936b-c131e22e5779 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:42.431788 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:42.446161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 411/825] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:07:42 2022] POST /identity/v3/endpoints => generated 346 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:07:42.446161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422b65a0-b911-4824-936b-c131e22e5779 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:42.502890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9db181f-02b6-4eec-bd21-f92415355d60 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:42.548531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3ddb76-d18f-4806-9279-9649e718c0e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:42.549250 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3ddb76-d18f-4806-9279-9649e718c0e7 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:42.549606 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3ddb76-d18f-4806-9279-9649e718c0e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:42.549993 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3ddb76-d18f-4806-9279-9649e718c0e7 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:42.550728 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3ddb76-d18f-4806-9279-9649e718c0e7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:42.551107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3ddb76-d18f-4806-9279-9649e718c0e7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:42.552804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3ddb76-d18f-4806-9279-9649e718c0e7 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:42.566589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 415/826] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:42.567447 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3ddb76-d18f-4806-9279-9649e718c0e7 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:42.592926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422b65a0-b911-4824-936b-c131e22e5779 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:42.642265 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80da59a1-8fb1-41cd-a7e7-ce655353ed0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:42.643350 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80da59a1-8fb1-41cd-a7e7-ce655353ed0f None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:42.643770 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80da59a1-8fb1-41cd-a7e7-ce655353ed0f None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:42.644591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80da59a1-8fb1-41cd-a7e7-ce655353ed0f None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:42.646184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80da59a1-8fb1-41cd-a7e7-ce655353ed0f None admin] RBAC: Adding query filter params (type=network) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:42.646622 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80da59a1-8fb1-41cd-a7e7-ce655353ed0f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:42.648807 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80da59a1-8fb1-41cd-a7e7-ce655353ed0f None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:42.654671 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 412/827] 10.222.0.26 () {56 vars in 1112 bytes} [Tue Sep 27 04:07:42 2022] GET /identity/v3/services?type=network => generated 350 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:42.655620 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80da59a1-8fb1-41cd-a7e7-ce655353ed0f None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:42.694609 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:42.699831 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3ddb76-d18f-4806-9279-9649e718c0e7 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:42.743054 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfd6f9d-040e-4791-ab64-a7cc2a446c67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:42.744170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfd6f9d-040e-4791-ab64-a7cc2a446c67 None admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:42.744654 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfd6f9d-040e-4791-ab64-a7cc2a446c67 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:42.745124 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfd6f9d-040e-4791-ab64-a7cc2a446c67 None admin] PATH_INFO: `/v3/projects/c830a3bfece6456486f4b094076b5364/users/5a2fcc8c9ac44dceadcc30d163bbfa0e/roles/20d83a8838004f60a32f8427577c7c35` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:42.763461 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfd6f9d-040e-4791-ab64-a7cc2a446c67 None admin] RBAC: Authorizing `identity:create_grant(project_id=c830a3bfece6456486f4b094076b5364, user_id=5a2fcc8c9ac44dceadcc30d163bbfa0e, role_id=20d83a8838004f60a32f8427577c7c35)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:42.765272 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfd6f9d-040e-4791-ab64-a7cc2a446c67 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:42.781078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 416/828] 10.222.0.26 () {58 vars in 1439 bytes} [Tue Sep 27 04:07:42 2022] PUT /identity/v3/projects/c830a3bfece6456486f4b094076b5364/users/5a2fcc8c9ac44dceadcc30d163bbfa0e/roles/20d83a8838004f60a32f8427577c7c35 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:07:42.782334 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfd6f9d-040e-4791-ab64-a7cc2a446c67 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:42.795826 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:42.918108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80da59a1-8fb1-41cd-a7e7-ce655353ed0f None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:42.932062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfd6f9d-040e-4791-ab64-a7cc2a446c67 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:42.994258 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2a66b1-1d38-42ef-b778-003d45545812 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:42.994904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2a66b1-1d38-42ef-b778-003d45545812 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:42.995177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2a66b1-1d38-42ef-b778-003d45545812 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:42.995447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2a66b1-1d38-42ef-b778-003d45545812 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:43.001129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d62f9a-7f14-4bf3-9e35-3129df4e8f0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:43.001771 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d62f9a-7f14-4bf3-9e35-3129df4e8f0d None admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:43.002054 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d62f9a-7f14-4bf3-9e35-3129df4e8f0d None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:43.002322 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d62f9a-7f14-4bf3-9e35-3129df4e8f0d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:43.002990 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d62f9a-7f14-4bf3-9e35-3129df4e8f0d None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:43.004605 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d62f9a-7f14-4bf3-9e35-3129df4e8f0d None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:43.007524 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2a66b1-1d38-42ef-b778-003d45545812 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:43.009154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2a66b1-1d38-42ef-b778-003d45545812 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:43.021925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-ab2a66b1-1d38-42ef-b778-003d45545812 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:43.021925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:43.021925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:43.021925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:43.021925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:43.021925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:43.021925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:43.021925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:43.021925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:43.021925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:43.021925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:43.021925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:43.021925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:43.021925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:43.021925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:43.023085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:43.023085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:43.023085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:43.023085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:43.023085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:43.023085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:43.023085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:43.023085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:43.023085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:43.023085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:43.023085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:43.023085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:43.023085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:43.023085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:43.023085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:43.023085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:43.023085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:43.024246 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:43.024246 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:43.024246 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:43.024246 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:43.024246 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:43.024246 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:43.024246 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:43.024246 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:43.027991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 413/829] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 371 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:43.032678 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2a66b1-1d38-42ef-b778-003d45545812 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:43.036803 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 417/830] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:07:42 2022] POST /identity/v3/endpoints => generated 337 bytes in 255 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:07:43.122682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:43.149105 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2a66b1-1d38-42ef-b778-003d45545812 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:43.193352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b102207-a6ec-441d-9b94-fe30e1cdce22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:43.193988 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b102207-a6ec-441d-9b94-fe30e1cdce22 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:43.194258 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b102207-a6ec-441d-9b94-fe30e1cdce22 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:43.194528 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b102207-a6ec-441d-9b94-fe30e1cdce22 None admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:43.195225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b102207-a6ec-441d-9b94-fe30e1cdce22 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:43.195525 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b102207-a6ec-441d-9b94-fe30e1cdce22 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:43.196969 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b102207-a6ec-441d-9b94-fe30e1cdce22 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:43.215791 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 414/831] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:43.218902 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d62f9a-7f14-4bf3-9e35-3129df4e8f0d None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:43.233122 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:43.344858 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d62f9a-7f14-4bf3-9e35-3129df4e8f0d None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:43.386233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7d5226-9290-4e90-8c8f-82ae35711e8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:43.386840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7d5226-9290-4e90-8c8f-82ae35711e8e None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:43.387117 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7d5226-9290-4e90-8c8f-82ae35711e8e None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:43.387388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7d5226-9290-4e90-8c8f-82ae35711e8e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:43.388117 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7d5226-9290-4e90-8c8f-82ae35711e8e None admin] RBAC: Adding query filter params (role.id=e6a31562a18649f0809202c36a8e18ef, scope.project.id=c830a3bfece6456486f4b094076b5364, user.id=a1586ee841cd44658e4d5feb25de7940) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:43.388422 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7d5226-9290-4e90-8c8f-82ae35711e8e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:43.389910 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7d5226-9290-4e90-8c8f-82ae35711e8e None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:43.396231 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 418/832] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:07:43 2022] GET /identity/v3/role_assignments?user.id=a1586ee841cd44658e4d5feb25de7940&scope.project.id=c830a3bfece6456486f4b094076b5364&role.id=e6a31562a18649f0809202c36a8e18ef => generated 608 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:43.480821 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a0db5e-b822-4860-b75c-0a25d2153560 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:43.481104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a0db5e-b822-4860-b75c-0a25d2153560 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:43.481356 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a0db5e-b822-4860-b75c-0a25d2153560 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:43.483024 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 415/833] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:43.488903 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2748e1d2-dbe4-4237-8e1e-bed47a5aae0c None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:43.489929 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2748e1d2-dbe4-4237-8e1e-bed47a5aae0c None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:43.490213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2748e1d2-dbe4-4237-8e1e-bed47a5aae0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:43.511600 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-2748e1d2-dbe4-4237-8e1e-bed47a5aae0c None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:43.512815 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-2748e1d2-dbe4-4237-8e1e-bed47a5aae0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:43.591228 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:43.597258 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 419/834] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:43 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:07:43.870452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a0db5e-b822-4860-b75c-0a25d2153560 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:43.981541 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a0db5e-b822-4860-b75c-0a25d2153560 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:44.011626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:44.017277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36921f13-088a-43e5-bc62-a06e7f62e51e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:44.018374 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36921f13-088a-43e5-bc62-a06e7f62e51e None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:44.019011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36921f13-088a-43e5-bc62-a06e7f62e51e None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:44.019667 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36921f13-088a-43e5-bc62-a06e7f62e51e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:44.030902 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36921f13-088a-43e5-bc62-a06e7f62e51e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:44.035806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-36921f13-088a-43e5-bc62-a06e7f62e51e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:07:44.035806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:44.035806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:44.035806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:44.035806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:44.035806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:44.035806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:44.035806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:44.035806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:44.035806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:44.035806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:44.035806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:44.035806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:44.035806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:44.035806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:44.035806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:44.037302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:44.037302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:44.037302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:44.037302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:44.037302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:44.037302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:44.037302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:44.037302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:44.037302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:44.037302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:44.037302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:44.037302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:44.037302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:44.037302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:44.037302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:44.037302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:44.038180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:44.038180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:44.038180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:44.038180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:44.038180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:44.038180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:44.038180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:44.038180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:44.038180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:44.038180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:44.038180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:07:44.038180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:44.038180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 416/835] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:07:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:44.043386 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2748e1d2-dbe4-4237-8e1e-bed47a5aae0c None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:44.144742 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2748e1d2-dbe4-4237-8e1e-bed47a5aae0c None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:44.180178 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32805e89-d952-4cc1-8f70-5ef2dadc2c47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:44.180778 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32805e89-d952-4cc1-8f70-5ef2dadc2c47 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:44.181053 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32805e89-d952-4cc1-8f70-5ef2dadc2c47 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:44.181319 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32805e89-d952-4cc1-8f70-5ef2dadc2c47 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:44.182007 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32805e89-d952-4cc1-8f70-5ef2dadc2c47 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:44.182283 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32805e89-d952-4cc1-8f70-5ef2dadc2c47 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:44.183622 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32805e89-d952-4cc1-8f70-5ef2dadc2c47 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:44.192282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 420/836] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:07:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:44.198970 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36921f13-088a-43e5-bc62-a06e7f62e51e None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:44.303153 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36921f13-088a-43e5-bc62-a06e7f62e51e None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:44.340161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403a588a-3de7-4b0c-9906-f52acd95a60c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:44.341127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403a588a-3de7-4b0c-9906-f52acd95a60c None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:44.341607 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403a588a-3de7-4b0c-9906-f52acd95a60c None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:44.342214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403a588a-3de7-4b0c-9906-f52acd95a60c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:44.352838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403a588a-3de7-4b0c-9906-f52acd95a60c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:44.354585 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403a588a-3de7-4b0c-9906-f52acd95a60c None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:44.370601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-403a588a-3de7-4b0c-9906-f52acd95a60c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:44.370601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:44.370601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:44.370601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:44.370601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:44.370601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:44.370601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:44.370601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:44.370601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:44.370601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:44.370601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:44.370601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:44.370601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:44.370601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:44.370601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:44.370601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:44.371546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:44.371546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:44.371546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:44.371546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:44.371546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:44.371546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:44.371546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:44.371546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:44.371546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:44.371546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:44.371546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:44.371546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:44.371546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:44.371546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:44.371546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:44.371546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:44.371546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:44.372472 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:44.372472 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:44.372472 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:44.374523 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 417/837] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:44.382602 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32805e89-d952-4cc1-8f70-5ef2dadc2c47 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:44.446976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:44.485655 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32805e89-d952-4cc1-8f70-5ef2dadc2c47 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:44.519132 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44aad60e-bfbb-42f6-a788-a5a124119495 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:44.519729 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44aad60e-bfbb-42f6-a788-a5a124119495 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:44.520006 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44aad60e-bfbb-42f6-a788-a5a124119495 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:44.520276 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44aad60e-bfbb-42f6-a788-a5a124119495 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:44.521546 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44aad60e-bfbb-42f6-a788-a5a124119495 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:44.521874 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44aad60e-bfbb-42f6-a788-a5a124119495 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:44.523213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44aad60e-bfbb-42f6-a788-a5a124119495 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:44.535214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 421/838] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:44.540666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403a588a-3de7-4b0c-9906-f52acd95a60c None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:44.586536 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:44.642317 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403a588a-3de7-4b0c-9906-f52acd95a60c None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:44.676035 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e6371f-5f8b-46ac-8c8b-6cea90de5142 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:44.676579 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e6371f-5f8b-46ac-8c8b-6cea90de5142 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:44.677390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e6371f-5f8b-46ac-8c8b-6cea90de5142 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:44.677656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e6371f-5f8b-46ac-8c8b-6cea90de5142 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:44.685298 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e6371f-5f8b-46ac-8c8b-6cea90de5142 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:44.686854 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e6371f-5f8b-46ac-8c8b-6cea90de5142 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:44.698461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-53e6371f-5f8b-46ac-8c8b-6cea90de5142 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 27 04:07:44.698461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:44.698461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:44.698461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:44.698461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:44.698461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:44.698461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:44.698461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:44.698461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:44.698461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:44.698461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:44.698461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:44.698461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:44.698461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:44.698461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:44.698461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:44.699390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:44.699390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:44.699390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:44.699390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:44.699390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:44.699390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:44.699390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:44.699390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:44.699390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:44.699390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:44.699390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:44.699390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:44.699390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:44.699390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:44.699390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:44.699390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:44.699390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:44.699390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:44.700341 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:44.700341 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:44.700341 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:44.700341 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:44.700341 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:44.700341 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:44.700341 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:44.700341 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:44.700341 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 27 04:07:44.700341 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:44.700341 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 418/839] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:07:44 2022] GET /identity/v3/users/swift => generated 83 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:44.705392 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44aad60e-bfbb-42f6-a788-a5a124119495 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:44.810416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44aad60e-bfbb-42f6-a788-a5a124119495 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:44.836041 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:44.847605 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac7ce25-98c5-437f-9b52-bf4de205ae65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:44.848202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac7ce25-98c5-437f-9b52-bf4de205ae65 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:44.848471 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac7ce25-98c5-437f-9b52-bf4de205ae65 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:44.848780 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac7ce25-98c5-437f-9b52-bf4de205ae65 None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:44.849718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac7ce25-98c5-437f-9b52-bf4de205ae65 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:44.850215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac7ce25-98c5-437f-9b52-bf4de205ae65 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:44.851622 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac7ce25-98c5-437f-9b52-bf4de205ae65 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:44.874106 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 422/840] 10.222.0.26 () {56 vars in 1135 bytes} [Tue Sep 27 04:07:44 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:44.875611 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e6371f-5f8b-46ac-8c8b-6cea90de5142 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:44.931036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66405607-95e3-4cfa-a551-3a4ec8cc079b None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:44.931228 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66405607-95e3-4cfa-a551-3a4ec8cc079b None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:44.931503 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66405607-95e3-4cfa-a551-3a4ec8cc079b None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:44.933091 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 423/841] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:07:44.938884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb0bc91-1c2d-45e5-b1c1-ec42438ccbf2 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:44.939159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb0bc91-1c2d-45e5-b1c1-ec42438ccbf2 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:44.939425 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb0bc91-1c2d-45e5-b1c1-ec42438ccbf2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:44.958318 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-1eb0bc91-1c2d-45e5-b1c1-ec42438ccbf2 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:44.959546 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-1eb0bc91-1c2d-45e5-b1c1-ec42438ccbf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:44.988949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e6371f-5f8b-46ac-8c8b-6cea90de5142 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:44.992869 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:45.029797 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749c4175-a4b7-40c1-824f-be5beb0e2966 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:45.030761 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749c4175-a4b7-40c1-824f-be5beb0e2966 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:45.031233 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749c4175-a4b7-40c1-824f-be5beb0e2966 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:45.031696 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749c4175-a4b7-40c1-824f-be5beb0e2966 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:45.041825 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 424/842] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:44 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 04:07:45.106731 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749c4175-a4b7-40c1-824f-be5beb0e2966 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:45.108579 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749c4175-a4b7-40c1-824f-be5beb0e2966 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:45.220208 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 419/843] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:07:44 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 04:07:45.226421 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb0bc91-1c2d-45e5-b1c1-ec42438ccbf2 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:45.307025 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b6161a-cd7a-4ef9-942e-e1d9ef532f89 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:45.307353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b6161a-cd7a-4ef9-942e-e1d9ef532f89 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:45.307612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b6161a-cd7a-4ef9-942e-e1d9ef532f89 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:45.309876 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 420/844] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:45.314975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808c451d-278f-4bb2-a41c-e4df5a17a60f None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:45.315263 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808c451d-278f-4bb2-a41c-e4df5a17a60f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:45.315541 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808c451d-278f-4bb2-a41c-e4df5a17a60f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:45.324645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:45.336190 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb0bc91-1c2d-45e5-b1c1-ec42438ccbf2 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:45.339638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-808c451d-278f-4bb2-a41c-e4df5a17a60f None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:45.341034 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-808c451d-278f-4bb2-a41c-e4df5a17a60f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:45.347031 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:45.372486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ac4260-6aa2-480a-8be6-8f9b0437d56f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:45.373105 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ac4260-6aa2-480a-8be6-8f9b0437d56f None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:45.373384 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ac4260-6aa2-480a-8be6-8f9b0437d56f None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:45.373650 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ac4260-6aa2-480a-8be6-8f9b0437d56f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:45.383829 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ac4260-6aa2-480a-8be6-8f9b0437d56f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:45.385252 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ac4260-6aa2-480a-8be6-8f9b0437d56f None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:45.400889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-18ac4260-6aa2-480a-8be6-8f9b0437d56f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:45.400889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:45.400889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:45.400889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:45.400889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:45.400889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:45.400889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:45.400889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:45.400889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:45.400889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:45.400889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:45.400889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:45.400889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:45.400889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:45.400889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:45.400889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:45.401869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:45.401869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:45.401869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:45.401869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:45.401869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:45.401869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:45.401869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:45.401869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:45.401869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:45.401869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:45.401869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:45.401869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:45.401869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:45.401869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:45.401869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:45.401869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:45.401869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:45.402786 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:45.402786 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:45.402786 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:45.406467 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 425/845] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:45.407311 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ac4260-6aa2-480a-8be6-8f9b0437d56f None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:45.421494 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 421/846] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:45 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:07:45.422640 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808c451d-278f-4bb2-a41c-e4df5a17a60f None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:45.535948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808c451d-278f-4bb2-a41c-e4df5a17a60f None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:45.546403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ac4260-6aa2-480a-8be6-8f9b0437d56f None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:45.575668 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af544fb-09cb-4bb8-9d46-bfde7a41301a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:45.576783 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af544fb-09cb-4bb8-9d46-bfde7a41301a None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:45.577295 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af544fb-09cb-4bb8-9d46-bfde7a41301a None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:45.577802 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af544fb-09cb-4bb8-9d46-bfde7a41301a None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:45.578660 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af544fb-09cb-4bb8-9d46-bfde7a41301a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:45.579235 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af544fb-09cb-4bb8-9d46-bfde7a41301a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:45.580977 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af544fb-09cb-4bb8-9d46-bfde7a41301a None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:45.583658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:45.591254 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fe73ee-1760-4e35-ba61-45a0f8b8f44d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:45.592201 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fe73ee-1760-4e35-ba61-45a0f8b8f44d None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:45.592489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fe73ee-1760-4e35-ba61-45a0f8b8f44d None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:45.592761 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fe73ee-1760-4e35-ba61-45a0f8b8f44d None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:45.593372 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16fe73ee-1760-4e35-ba61-45a0f8b8f44d None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:45.595622 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16fe73ee-1760-4e35-ba61-45a0f8b8f44d None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:45.599705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 422/847] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:45.609817 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af544fb-09cb-4bb8-9d46-bfde7a41301a None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:45.610035 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-16fe73ee-1760-4e35-ba61-45a0f8b8f44d None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 27 04:07:45.610035 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:45.610035 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:07:45.610035 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:07:45.610035 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:07:45.610035 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:07:45.610035 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:07:45.610035 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:45.610035 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 27 04:07:45.610035 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:45.610035 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:07:45.610035 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:45.610035 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:45.610035 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:45.610035 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:45.610035 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:45.610035 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:45.611162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:45.611162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:45.611162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:45.611162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:45.611162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:45.611162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:45.611162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:07:45.611162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:07:45.611162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:07:45.611162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:07:45.611162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:45.611162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:45.611162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:45.611162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:45.611162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:45.611162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:45.611989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:45.611989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:45.611989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:45.611989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:45.611989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:45.611989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:45.611989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:45.611989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:45.611989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:45.611989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:45.611989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:07:45.611989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:45.611989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 27 04:07:45.611989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:45.613856 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 426/848] 10.222.0.26 () {56 vars in 1114 bytes} [Tue Sep 27 04:07:45 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:45.616040 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fe73ee-1760-4e35-ba61-45a0f8b8f44d None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:45.664181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:45.729090 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af544fb-09cb-4bb8-9d46-bfde7a41301a None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:45.744510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fe73ee-1760-4e35-ba61-45a0f8b8f44d None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:45.762021 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddc032b-6452-4977-b873-31951d3ab1b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:45.762646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddc032b-6452-4977-b873-31951d3ab1b9 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:45.762923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddc032b-6452-4977-b873-31951d3ab1b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:45.763196 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddc032b-6452-4977-b873-31951d3ab1b9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:45.769642 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ddc032b-6452-4977-b873-31951d3ab1b9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:45.771684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ddc032b-6452-4977-b873-31951d3ab1b9 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:45.777636 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1819644c-6029-4de2-9163-3e562cf521a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:45.778253 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1819644c-6029-4de2-9163-3e562cf521a1 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:45.778527 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1819644c-6029-4de2-9163-3e562cf521a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:45.778795 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1819644c-6029-4de2-9163-3e562cf521a1 None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:45.779440 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1819644c-6029-4de2-9163-3e562cf521a1 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:45.779736 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1819644c-6029-4de2-9163-3e562cf521a1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:45.781054 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1819644c-6029-4de2-9163-3e562cf521a1 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:45.782956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-6ddc032b-6452-4977-b873-31951d3ab1b9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:45.782956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:45.782956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:45.782956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:45.782956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:45.782956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:45.782956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:45.782956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:45.782956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:45.782956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:45.782956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:45.782956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:45.782956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:45.782956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:45.782956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:45.784002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:45.784002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:45.784002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:45.784002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:45.784002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:45.784002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:45.784002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:45.784002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:45.784002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:45.784002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:45.784002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:45.784002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:45.784002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:45.784002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:45.784002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:45.784002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:45.784002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:45.785090 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:45.785090 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:45.785090 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:45.785090 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:45.785090 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:45.785090 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:45.785090 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:45.785090 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:45.785090 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 423/849] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:45.785090 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddc032b-6452-4977-b873-31951d3ab1b9 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:45.792344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 427/850] 10.222.0.26 () {56 vars in 1114 bytes} [Tue Sep 27 04:07:45 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:07:45.792344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a928bd-dc8b-4721-8278-72a488f5cfac None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:45.792344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a928bd-dc8b-4721-8278-72a488f5cfac None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:45.792344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a928bd-dc8b-4721-8278-72a488f5cfac None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:45.792847 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 428/851] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:45.793550 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a928bd-dc8b-4721-8278-72a488f5cfac None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:45.839175 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:45.882334 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddc032b-6452-4977-b873-31951d3ab1b9 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:45.885629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:45.896768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a928bd-dc8b-4721-8278-72a488f5cfac None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:45.916499 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7633c3-138d-4bd7-9888-897b9789e739 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:45.917124 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7633c3-138d-4bd7-9888-897b9789e739 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:45.917404 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7633c3-138d-4bd7-9888-897b9789e739 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:45.917678 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7633c3-138d-4bd7-9888-897b9789e739 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:45.927572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7633c3-138d-4bd7-9888-897b9789e739 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:45.928971 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b7f27d-3262-43a0-a11a-836147323210 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:45.929560 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b7f27d-3262-43a0-a11a-836147323210 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:45.929879 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b7f27d-3262-43a0-a11a-836147323210 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:45.930155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b7f27d-3262-43a0-a11a-836147323210 None admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:45.930926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b7f27d-3262-43a0-a11a-836147323210 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:45.931230 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b7f27d-3262-43a0-a11a-836147323210 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:45.932643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b7f27d-3262-43a0-a11a-836147323210 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:45.932766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-6f7633c3-138d-4bd7-9888-897b9789e739 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:45.932766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:45.932766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:45.932766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:45.932766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:45.932766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:45.932766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:45.932766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:45.932766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:45.932766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:45.932766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:45.932766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:45.932766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:45.932766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:45.932766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:45.932766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:45.933836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:45.933836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:45.933836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:45.933836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:45.933836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:45.933836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:45.933836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:45.933836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:45.933836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:45.933836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:45.933836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:45.933836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:45.933836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:45.933836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:45.933836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:45.933836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:45.934694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:45.934694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:45.934694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:45.934694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:45.934694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:45.934694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:45.934694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:45.934694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:45.934694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:45.934694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:45.934694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:07:45.934694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:45.934694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 424/852] 10.222.0.26 () {56 vars in 1102 bytes} [Tue Sep 27 04:07:45 2022] GET /identity/v3/roles/service => generated 85 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:45.934694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7633c3-138d-4bd7-9888-897b9789e739 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:45.946986 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 429/853] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:45.948223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0bc8d4-81ae-43bb-9b3d-16d85d48cc16 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:45.948506 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0bc8d4-81ae-43bb-9b3d-16d85d48cc16 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:45.948778 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0bc8d4-81ae-43bb-9b3d-16d85d48cc16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:45.966476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-9f0bc8d4-81ae-43bb-9b3d-16d85d48cc16 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:45.967653 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-9f0bc8d4-81ae-43bb-9b3d-16d85d48cc16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:46.044657 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 430/854] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:45 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:07:46.044657 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0bc8d4-81ae-43bb-9b3d-16d85d48cc16 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:46.044901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7633c3-138d-4bd7-9888-897b9789e739 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:46.052455 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:46.075057 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35afb1f-19b7-43fb-95b3-8e20137ec206 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:46.075658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35afb1f-19b7-43fb-95b3-8e20137ec206 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:46.075938 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35afb1f-19b7-43fb-95b3-8e20137ec206 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:46.076212 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35afb1f-19b7-43fb-95b3-8e20137ec206 None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:46.076840 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35afb1f-19b7-43fb-95b3-8e20137ec206 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:46.077140 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35afb1f-19b7-43fb-95b3-8e20137ec206 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:46.078516 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35afb1f-19b7-43fb-95b3-8e20137ec206 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:46.082818 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 425/855] 10.222.0.26 () {56 vars in 1114 bytes} [Tue Sep 27 04:07:45 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:46.083613 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35afb1f-19b7-43fb-95b3-8e20137ec206 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:46.137391 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:46.158277 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0bc8d4-81ae-43bb-9b3d-16d85d48cc16 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:46.198198 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1cbad0-ef91-4358-b5a9-630bcf6c1dec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:46.198887 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1cbad0-ef91-4358-b5a9-630bcf6c1dec None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:46.199168 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1cbad0-ef91-4358-b5a9-630bcf6c1dec None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:46.199435 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1cbad0-ef91-4358-b5a9-630bcf6c1dec None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:46.200115 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1cbad0-ef91-4358-b5a9-630bcf6c1dec None admin] RBAC: Adding query filter params (name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:46.200415 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1cbad0-ef91-4358-b5a9-630bcf6c1dec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:46.202171 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1cbad0-ef91-4358-b5a9-630bcf6c1dec None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:46.212934 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 431/856] 10.222.0.26 () {56 vars in 1103 bytes} [Tue Sep 27 04:07:46 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:46.219455 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1cbad0-ef91-4358-b5a9-630bcf6c1dec None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:46.232573 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35afb1f-19b7-43fb-95b3-8e20137ec206 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:46.280224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123bcc32-5af0-41d7-9af6-b8e4638ce20a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:46.281345 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123bcc32-5af0-41d7-9af6-b8e4638ce20a None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:46.281835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123bcc32-5af0-41d7-9af6-b8e4638ce20a None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:46.282677 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123bcc32-5af0-41d7-9af6-b8e4638ce20a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:46.283596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123bcc32-5af0-41d7-9af6-b8e4638ce20a None admin] RBAC: Adding query filter params (role.id=e6a31562a18649f0809202c36a8e18ef, scope.project.id=c830a3bfece6456486f4b094076b5364, user.id=181dd5dc720c4a97860852062b7b19c3) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:46.283914 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123bcc32-5af0-41d7-9af6-b8e4638ce20a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:46.285338 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123bcc32-5af0-41d7-9af6-b8e4638ce20a None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:46.300198 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 426/857] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:07:46 2022] GET /identity/v3/role_assignments?user.id=181dd5dc720c4a97860852062b7b19c3&scope.project.id=c830a3bfece6456486f4b094076b5364&role.id=e6a31562a18649f0809202c36a8e18ef => generated 608 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:46.300198 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123bcc32-5af0-41d7-9af6-b8e4638ce20a None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:46.343710 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1cbad0-ef91-4358-b5a9-630bcf6c1dec None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:46.351300 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:46.390385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c81164e-afce-49fb-be88-41e67f1bcaaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:46.391343 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c81164e-afce-49fb-be88-41e67f1bcaaa None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:46.391628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c81164e-afce-49fb-be88-41e67f1bcaaa None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:46.391904 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c81164e-afce-49fb-be88-41e67f1bcaaa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:46.407821 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c81164e-afce-49fb-be88-41e67f1bcaaa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:46.410017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c81164e-afce-49fb-be88-41e67f1bcaaa None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:46.412854 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:46.431181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-9c81164e-afce-49fb-be88-41e67f1bcaaa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:46.431181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:46.431181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:46.431181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:46.431181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:46.431181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:46.431181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:46.431181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:46.431181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:46.431181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:46.431181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:46.431181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:46.431181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:46.431181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:46.431181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:46.431181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:46.432407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:46.432407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:46.432407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:46.432407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:46.432407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:46.432407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:46.432407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:46.432407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:46.432407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:46.432407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:46.432407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:46.432407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:46.432407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:46.432407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:46.432407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:46.432407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:46.432407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:46.433496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:46.433496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:46.433496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:46.433496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 432/858] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:07:46.438981 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c81164e-afce-49fb-be88-41e67f1bcaaa None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:46.446379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123bcc32-5af0-41d7-9af6-b8e4638ce20a None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:46.485233 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a99df67-14ff-4485-91cc-217ae0b3fc1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:46.485855 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a99df67-14ff-4485-91cc-217ae0b3fc1a None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:46.486139 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a99df67-14ff-4485-91cc-217ae0b3fc1a None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:46.486413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a99df67-14ff-4485-91cc-217ae0b3fc1a None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:46.487006 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a99df67-14ff-4485-91cc-217ae0b3fc1a None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:46.488515 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a99df67-14ff-4485-91cc-217ae0b3fc1a None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:46.498929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-1a99df67-14ff-4485-91cc-217ae0b3fc1a None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 27 04:07:46.498929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:46.498929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:07:46.498929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:07:46.498929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:07:46.498929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:07:46.498929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:07:46.498929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:46.498929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 27 04:07:46.498929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:46.498929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:07:46.498929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:46.498929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:46.498929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:46.498929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:46.498929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:46.499926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:46.499926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:46.499926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:46.499926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:46.499926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:46.499926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:46.499926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:46.499926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:07:46.499926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:07:46.499926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:07:46.499926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:07:46.499926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:46.499926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:46.499926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:46.499926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:46.499926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:46.499926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:46.500838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:46.500838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:46.500838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:46.500838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:46.500838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:46.500838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:46.500838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:46.500838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:46.500838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:46.500838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:46.500838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:07:46.500838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:46.500838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 27 04:07:46.500838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:46.500838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 427/859] 10.222.0.26 () {56 vars in 1132 bytes} [Tue Sep 27 04:07:46 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:46.509777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a99df67-14ff-4485-91cc-217ae0b3fc1a None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:46.588387 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c81164e-afce-49fb-be88-41e67f1bcaaa None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:46.627797 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a99df67-14ff-4485-91cc-217ae0b3fc1a None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:46.628304 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3867fab6-9e86-414d-8a60-1bfb26958954 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:46.629083 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3867fab6-9e86-414d-8a60-1bfb26958954 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:46.629800 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3867fab6-9e86-414d-8a60-1bfb26958954 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:46.630273 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3867fab6-9e86-414d-8a60-1bfb26958954 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:46.631092 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3867fab6-9e86-414d-8a60-1bfb26958954 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:46.631397 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3867fab6-9e86-414d-8a60-1bfb26958954 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:46.632745 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3867fab6-9e86-414d-8a60-1bfb26958954 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:46.636154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:46.649873 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 433/860] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:46.652276 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3867fab6-9e86-414d-8a60-1bfb26958954 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:46.668196 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac8b60b-a2a4-4daa-a43e-fe7aec0efb6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:46.669144 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac8b60b-a2a4-4daa-a43e-fe7aec0efb6e None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:46.669603 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac8b60b-a2a4-4daa-a43e-fe7aec0efb6e None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:46.670100 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac8b60b-a2a4-4daa-a43e-fe7aec0efb6e None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:46.670909 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac8b60b-a2a4-4daa-a43e-fe7aec0efb6e None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:46.671392 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac8b60b-a2a4-4daa-a43e-fe7aec0efb6e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:46.673222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac8b60b-a2a4-4daa-a43e-fe7aec0efb6e None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:46.679415 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 428/861] 10.222.0.26 () {56 vars in 1126 bytes} [Tue Sep 27 04:07:46 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:46.687598 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac8b60b-a2a4-4daa-a43e-fe7aec0efb6e None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:46.722711 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:46.786631 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3867fab6-9e86-414d-8a60-1bfb26958954 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:46.792324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac8b60b-a2a4-4daa-a43e-fe7aec0efb6e None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:46.822219 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412ba690-91c7-49d0-a817-9112cf617452 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:46.823006 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412ba690-91c7-49d0-a817-9112cf617452 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:46.823283 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412ba690-91c7-49d0-a817-9112cf617452 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:46.823552 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412ba690-91c7-49d0-a817-9112cf617452 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:46.830944 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412ba690-91c7-49d0-a817-9112cf617452 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:46.832462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412ba690-91c7-49d0-a817-9112cf617452 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:46.835241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39687537-8b22-4b1e-a14d-71d6eb86d89b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:46.836501 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39687537-8b22-4b1e-a14d-71d6eb86d89b None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:46.836849 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39687537-8b22-4b1e-a14d-71d6eb86d89b None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:46.837194 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39687537-8b22-4b1e-a14d-71d6eb86d89b None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:46.837932 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39687537-8b22-4b1e-a14d-71d6eb86d89b None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:46.838305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39687537-8b22-4b1e-a14d-71d6eb86d89b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:46.839991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39687537-8b22-4b1e-a14d-71d6eb86d89b None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:46.845058 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 429/862] 10.222.0.26 () {56 vars in 1126 bytes} [Tue Sep 27 04:07:46 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:46.854431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-412ba690-91c7-49d0-a817-9112cf617452 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 27 04:07:46.854431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:46.854431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:46.854431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:46.854431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:46.854431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:46.854431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:46.854431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:46.854431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:46.854431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:46.854431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:46.854431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:46.854431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:46.854431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:46.854431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:46.854431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:46.855662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:46.855662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:46.855662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:46.855662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:46.855662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:46.855662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:46.855662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:46.855662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:46.855662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:46.855662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:46.855662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:46.855662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:46.855662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:46.855662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:46.855662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:46.855662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:46.855662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:46.855662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:46.856683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:46.856683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:46.856683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:46.856683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:46.856683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:46.856683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:46.856683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:46.856683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:46.856683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 27 04:07:46.856683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:46.856683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 434/863] 10.222.0.26 () {56 vars in 1117 bytes} [Tue Sep 27 04:07:46 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:07:46.861550 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39687537-8b22-4b1e-a14d-71d6eb86d89b None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:46.902091 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:46.979269 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39687537-8b22-4b1e-a14d-71d6eb86d89b None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:47.017826 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4658b2-db96-42b6-a849-b17cb6052c53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:47.018812 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4658b2-db96-42b6-a849-b17cb6052c53 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:47.019322 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4658b2-db96-42b6-a849-b17cb6052c53 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:47.019795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4658b2-db96-42b6-a849-b17cb6052c53 None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:47.021267 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba4658b2-db96-42b6-a849-b17cb6052c53 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:47.021803 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba4658b2-db96-42b6-a849-b17cb6052c53 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:47.023458 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba4658b2-db96-42b6-a849-b17cb6052c53 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:47.044867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 430/864] 10.222.0.26 () {56 vars in 1149 bytes} [Tue Sep 27 04:07:46 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:47.051362 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412ba690-91c7-49d0-a817-9112cf617452 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:47.159039 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412ba690-91c7-49d0-a817-9112cf617452 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:47.173513 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:47.194673 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2ac0b7-353d-4d2b-851e-9840ad131f81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:47.195277 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2ac0b7-353d-4d2b-851e-9840ad131f81 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:47.195551 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2ac0b7-353d-4d2b-851e-9840ad131f81 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:47.195820 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2ac0b7-353d-4d2b-851e-9840ad131f81 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:47.264351 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2ac0b7-353d-4d2b-851e-9840ad131f81 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:47.265841 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2ac0b7-353d-4d2b-851e-9840ad131f81 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:47.370202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 435/865] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:07:47 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 04:07:47.376283 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4658b2-db96-42b6-a849-b17cb6052c53 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:47.485837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:47.495440 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4658b2-db96-42b6-a849-b17cb6052c53 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:47.533859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122da77c-19fc-4f33-8900-11d08ae485fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:47.534491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122da77c-19fc-4f33-8900-11d08ae485fa None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:47.534769 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122da77c-19fc-4f33-8900-11d08ae485fa None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:47.535042 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122da77c-19fc-4f33-8900-11d08ae485fa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:47.545375 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122da77c-19fc-4f33-8900-11d08ae485fa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:47.546837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122da77c-19fc-4f33-8900-11d08ae485fa None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:47.562116 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-122da77c-19fc-4f33-8900-11d08ae485fa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:47.562116 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:47.562116 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:47.562116 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:47.562116 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:47.562116 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:47.562116 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:47.562116 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:47.562116 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:47.562116 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:47.562116 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:47.562116 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:47.562116 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:47.562116 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:47.562116 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:47.562116 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:47.563068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:47.563068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:47.563068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:47.563068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:47.563068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:47.563068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:47.563068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:47.563068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:47.563068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:47.563068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:47.563068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:47.563068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:47.563068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:47.563068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:47.563068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:47.563068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:47.563068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:47.563941 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:47.563941 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:47.563941 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:47.563941 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 431/866] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:47.570882 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2ac0b7-353d-4d2b-851e-9840ad131f81 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:47.586335 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:47.674305 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2ac0b7-353d-4d2b-851e-9840ad131f81 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:47.712822 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f92b4a-2a1f-4d96-a721-759a5ea9aec8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:47.713421 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f92b4a-2a1f-4d96-a721-759a5ea9aec8 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:47.713693 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f92b4a-2a1f-4d96-a721-759a5ea9aec8 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:47.713988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f92b4a-2a1f-4d96-a721-759a5ea9aec8 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:47.714631 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f92b4a-2a1f-4d96-a721-759a5ea9aec8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:47.714924 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f92b4a-2a1f-4d96-a721-759a5ea9aec8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:47.716265 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f92b4a-2a1f-4d96-a721-759a5ea9aec8 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:47.728903 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 436/867] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:47.734307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122da77c-19fc-4f33-8900-11d08ae485fa None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:47.847193 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122da77c-19fc-4f33-8900-11d08ae485fa None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:47.883998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00161126-be50-4765-b92e-428adba7c2f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:47.885007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00161126-be50-4765-b92e-428adba7c2f5 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:47.885498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00161126-be50-4765-b92e-428adba7c2f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:47.886021 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00161126-be50-4765-b92e-428adba7c2f5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:47.892954 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00161126-be50-4765-b92e-428adba7c2f5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:47.894663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00161126-be50-4765-b92e-428adba7c2f5 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:47.906154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-00161126-be50-4765-b92e-428adba7c2f5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:47.906154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:47.906154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:47.906154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:47.906154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:47.906154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:47.906154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:47.906154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:47.906154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:47.906154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:47.906154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:47.906154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:47.906154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:47.906154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:47.906154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:47.907446 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:47.907446 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:47.907446 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:47.907446 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:47.907446 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:47.907446 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:47.907446 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:47.907446 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:47.907446 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:47.907446 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:47.907446 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:47.907446 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:47.907446 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:47.907446 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:47.907446 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:47.907446 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:47.907446 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:47.908328 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:47.908328 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:47.908328 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:47.908328 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:47.908328 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:47.908328 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:47.908328 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:47.908328 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:47.910594 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 432/868] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:47.918814 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f92b4a-2a1f-4d96-a721-759a5ea9aec8 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:47.996020 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:48.022618 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f92b4a-2a1f-4d96-a721-759a5ea9aec8 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:48.057535 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce39b40-29bd-4f6c-9006-01600de625e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:48.058167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce39b40-29bd-4f6c-9006-01600de625e3 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:48.058443 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce39b40-29bd-4f6c-9006-01600de625e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:48.058709 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce39b40-29bd-4f6c-9006-01600de625e3 None admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:48.059393 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce39b40-29bd-4f6c-9006-01600de625e3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:48.059692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce39b40-29bd-4f6c-9006-01600de625e3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:48.061174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce39b40-29bd-4f6c-9006-01600de625e3 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:48.073771 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 437/869] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:48.079697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00161126-be50-4765-b92e-428adba7c2f5 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:48.083781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:48.191122 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00161126-be50-4765-b92e-428adba7c2f5 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:48.231705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53f3ca1-2391-4778-9cb1-cd36cf15ce63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:48.232391 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53f3ca1-2391-4778-9cb1-cd36cf15ce63 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:48.232663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53f3ca1-2391-4778-9cb1-cd36cf15ce63 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:48.232929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53f3ca1-2391-4778-9cb1-cd36cf15ce63 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:48.233707 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53f3ca1-2391-4778-9cb1-cd36cf15ce63 None admin] RBAC: Adding query filter params (role.id=20d83a8838004f60a32f8427577c7c35, scope.project.id=c830a3bfece6456486f4b094076b5364, user.id=5a2fcc8c9ac44dceadcc30d163bbfa0e) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:48.234043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53f3ca1-2391-4778-9cb1-cd36cf15ce63 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:48.235478 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53f3ca1-2391-4778-9cb1-cd36cf15ce63 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:48.243143 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 433/870] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:07:48 2022] GET /identity/v3/role_assignments?user.id=5a2fcc8c9ac44dceadcc30d163bbfa0e&scope.project.id=c830a3bfece6456486f4b094076b5364&role.id=20d83a8838004f60a32f8427577c7c35 => generated 608 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:48.329986 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35004d0-6b5d-449b-80c8-d228c7413c4c None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:48.330329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35004d0-6b5d-449b-80c8-d228c7413c4c None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:48.330551 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35004d0-6b5d-449b-80c8-d228c7413c4c None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:48.332189 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 438/871] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:48.338509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6eb0177-2df4-4ea0-b98e-41dee829abcb None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:48.338828 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6eb0177-2df4-4ea0-b98e-41dee829abcb None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:48.339060 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6eb0177-2df4-4ea0-b98e-41dee829abcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:48.358816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-b6eb0177-2df4-4ea0-b98e-41dee829abcb None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:48.360024 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-b6eb0177-2df4-4ea0-b98e-41dee829abcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:48.432778 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:48.439048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 434/872] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:48 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:07:48.708957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35004d0-6b5d-449b-80c8-d228c7413c4c None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:48.711227 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c034c7d9-3f76-4485-81c5-aa09f4ed92d5 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:48.711510 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c034c7d9-3f76-4485-81c5-aa09f4ed92d5 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:48.711790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c034c7d9-3f76-4485-81c5-aa09f4ed92d5 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:48.713364 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 435/873] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:48.723467 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872c94ee-392e-4d57-9591-794e3e676e2c None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:48.723794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872c94ee-392e-4d57-9591-794e3e676e2c None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:48.724345 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872c94ee-392e-4d57-9591-794e3e676e2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:48.746542 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-872c94ee-392e-4d57-9591-794e3e676e2c None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:48.748740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-872c94ee-392e-4d57-9591-794e3e676e2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:48.824413 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35004d0-6b5d-449b-80c8-d228c7413c4c None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:48.842072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 436/874] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:48 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 04:07:48.859555 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc27d9a7-4e03-40d2-9f8c-5c53143334e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:48.860445 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc27d9a7-4e03-40d2-9f8c-5c53143334e6 None admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:48.860950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc27d9a7-4e03-40d2-9f8c-5c53143334e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:48.861424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc27d9a7-4e03-40d2-9f8c-5c53143334e6 None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:48.862343 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc27d9a7-4e03-40d2-9f8c-5c53143334e6 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:48.863894 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc27d9a7-4e03-40d2-9f8c-5c53143334e6 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:48.867561 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:48.887381 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 439/875] 10.222.0.26 () {60 vars in 1141 bytes} [Tue Sep 27 04:07:48 2022] POST /identity/v3/services => generated 251 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:07:49.121933 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872c94ee-392e-4d57-9591-794e3e676e2c None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:49.195943 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500f404f-8d6b-47cb-ba40-7a9b65afa881 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:49.196266 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500f404f-8d6b-47cb-ba40-7a9b65afa881 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:49.196488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500f404f-8d6b-47cb-ba40-7a9b65afa881 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:49.198086 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 440/876] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:49.202768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499232b3-2cdb-46a3-a8c5-5ce682ae4aff None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:49.203045 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499232b3-2cdb-46a3-a8c5-5ce682ae4aff None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:49.203318 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499232b3-2cdb-46a3-a8c5-5ce682ae4aff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:49.224776 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-499232b3-2cdb-46a3-a8c5-5ce682ae4aff None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:49.225997 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-499232b3-2cdb-46a3-a8c5-5ce682ae4aff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:49.230717 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872c94ee-392e-4d57-9591-794e3e676e2c None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:49.284707 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212eebab-83da-46ac-a49d-7dd152e7d85d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:49.285366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212eebab-83da-46ac-a49d-7dd152e7d85d None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:49.285873 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212eebab-83da-46ac-a49d-7dd152e7d85d None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:49.286179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212eebab-83da-46ac-a49d-7dd152e7d85d None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:49.287180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212eebab-83da-46ac-a49d-7dd152e7d85d None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:49.288546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212eebab-83da-46ac-a49d-7dd152e7d85d None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:49.299288 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:49.301636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-212eebab-83da-46ac-a49d-7dd152e7d85d None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 04:07:49.301636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:49.301636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:07:49.301636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:07:49.301636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:07:49.301636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:07:49.301636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:07:49.301636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:49.301636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 04:07:49.301636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:49.301636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:07:49.301636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:49.301636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:49.301636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:49.301636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:49.301636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:49.301636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:49.303360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:49.303360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:49.303360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:49.303360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:49.303360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:49.303360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:49.303360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:07:49.303360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:07:49.303360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:07:49.303360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:07:49.303360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:49.303360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:49.303360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:49.303360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:49.303360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:49.303360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:49.304244 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:49.304244 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:49.304244 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:49.304244 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:49.304244 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:49.304244 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:49.304244 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:49.304244 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:49.304244 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:49.304244 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:49.304244 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:07:49.304244 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:49.304244 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 04:07:49.304244 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:49.308287 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 437/877] 10.222.0.26 () {56 vars in 1126 bytes} [Tue Sep 27 04:07:49 2022] GET /identity/v3/services/object-store => generated 93 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:49.309215 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212eebab-83da-46ac-a49d-7dd152e7d85d None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:49.310938 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 441/878] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:49 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:07:49.430137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212eebab-83da-46ac-a49d-7dd152e7d85d None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:49.464647 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e675e4c7-f930-48ea-814d-fa0ff4be52cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:49.465573 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e675e4c7-f930-48ea-814d-fa0ff4be52cd None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:49.466079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e675e4c7-f930-48ea-814d-fa0ff4be52cd None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:49.466554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e675e4c7-f930-48ea-814d-fa0ff4be52cd None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:49.467397 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e675e4c7-f930-48ea-814d-fa0ff4be52cd None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:49.467917 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e675e4c7-f930-48ea-814d-fa0ff4be52cd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:49.469481 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e675e4c7-f930-48ea-814d-fa0ff4be52cd None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:49.475085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 438/879] 10.222.0.26 () {56 vars in 1122 bytes} [Tue Sep 27 04:07:49 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:49.485799 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499232b3-2cdb-46a3-a8c5-5ce682ae4aff None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:49.581956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499232b3-2cdb-46a3-a8c5-5ce682ae4aff None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:49.595568 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e675e4c7-f930-48ea-814d-fa0ff4be52cd None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:49.608098 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:49.621182 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8323883c-4814-40bf-a8f3-f3861317b72c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:49.621813 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8323883c-4814-40bf-a8f3-f3861317b72c None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:49.622087 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8323883c-4814-40bf-a8f3-f3861317b72c None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:49.622360 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8323883c-4814-40bf-a8f3-f3861317b72c None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:49.622993 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8323883c-4814-40bf-a8f3-f3861317b72c None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:49.623291 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8323883c-4814-40bf-a8f3-f3861317b72c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:49.624650 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8323883c-4814-40bf-a8f3-f3861317b72c None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:49.630751 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 442/880] 10.222.0.26 () {56 vars in 1122 bytes} [Tue Sep 27 04:07:49 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:49.660660 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:49.725546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e675e4c7-f930-48ea-814d-fa0ff4be52cd None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:49.763737 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ab06f0-2319-405b-b7a4-9d0f2c709cc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:49.764356 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ab06f0-2319-405b-b7a4-9d0f2c709cc8 None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:49.764630 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ab06f0-2319-405b-b7a4-9d0f2c709cc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:49.764899 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ab06f0-2319-405b-b7a4-9d0f2c709cc8 None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:49.765569 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ab06f0-2319-405b-b7a4-9d0f2c709cc8 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:49.766966 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ab06f0-2319-405b-b7a4-9d0f2c709cc8 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:49.789962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 439/881] 10.222.0.26 () {60 vars in 1141 bytes} [Tue Sep 27 04:07:49 2022] POST /identity/v3/services => generated 269 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:07:50.570016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf0c54a-5f4e-4b5a-80a7-9a0b2a7a7d83 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:50.570529 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf0c54a-5f4e-4b5a-80a7-9a0b2a7a7d83 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:50.570802 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf0c54a-5f4e-4b5a-80a7-9a0b2a7a7d83 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:50.572241 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 443/882] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:50.577624 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e6721a-2460-4eaa-b3cb-9f0111cd618f None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:50.578065 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e6721a-2460-4eaa-b3cb-9f0111cd618f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:50.578239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e6721a-2460-4eaa-b3cb-9f0111cd618f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:50.596430 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-91e6721a-2460-4eaa-b3cb-9f0111cd618f None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:50.597611 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-91e6721a-2460-4eaa-b3cb-9f0111cd618f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:50.675316 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 440/883] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:50 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 04:07:50.946027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf0c54a-5f4e-4b5a-80a7-9a0b2a7a7d83 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:51.050002 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf0c54a-5f4e-4b5a-80a7-9a0b2a7a7d83 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:51.086198 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ccd43e1-03e3-4f46-909d-9c8ff98d2261 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:51.086823 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ccd43e1-03e3-4f46-909d-9c8ff98d2261 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:51.087098 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ccd43e1-03e3-4f46-909d-9c8ff98d2261 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:51.087364 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ccd43e1-03e3-4f46-909d-9c8ff98d2261 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:51.097540 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ccd43e1-03e3-4f46-909d-9c8ff98d2261 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:51.102834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-1ccd43e1-03e3-4f46-909d-9c8ff98d2261 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 27 04:07:51.102834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:51.102834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:51.102834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:51.102834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:51.102834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:51.102834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:51.102834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:51.102834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:51.102834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:51.102834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:51.102834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:51.102834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:51.102834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:51.102834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:51.102834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:51.103763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:51.103763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:51.103763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:51.103763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:51.103763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:51.103763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:51.103763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:51.103763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:51.103763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:51.103763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:51.103763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:51.103763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:51.103763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:51.103763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:51.103763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:51.103763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:51.104603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:51.104603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:51.104603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:51.104603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:51.104603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:51.104603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:51.104603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:51.104603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:51.104603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:51.104603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:51.104603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 27 04:07:51.104603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:51.104603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 444/884] 10.222.0.26 () {56 vars in 1120 bytes} [Tue Sep 27 04:07:50 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:51.109832 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e6721a-2460-4eaa-b3cb-9f0111cd618f None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:51.149849 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:51.156453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac596c21-bd57-4f22-bab5-bb513c09df1f None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:51.156734 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac596c21-bd57-4f22-bab5-bb513c09df1f None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:51.157004 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac596c21-bd57-4f22-bab5-bb513c09df1f None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:51.159041 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 445/885] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:07:51.164953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e14f45-8541-4539-bdfe-f62247c14b86 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:51.166136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e14f45-8541-4539-bdfe-f62247c14b86 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:51.166902 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e14f45-8541-4539-bdfe-f62247c14b86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:51.186272 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-60e14f45-8541-4539-bdfe-f62247c14b86 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:51.187462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-60e14f45-8541-4539-bdfe-f62247c14b86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:51.215817 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e6721a-2460-4eaa-b3cb-9f0111cd618f None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:51.237665 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:51.255873 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08b2df8-f938-4584-a116-d2eba252c122 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:51.256809 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08b2df8-f938-4584-a116-d2eba252c122 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:51.257309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08b2df8-f938-4584-a116-d2eba252c122 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:51.257904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08b2df8-f938-4584-a116-d2eba252c122 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:51.258878 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08b2df8-f938-4584-a116-d2eba252c122 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:51.259377 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08b2df8-f938-4584-a116-d2eba252c122 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:51.260930 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08b2df8-f938-4584-a116-d2eba252c122 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:51.269793 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 446/886] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:51 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 04:07:51.273479 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 441/887] 10.222.0.26 () {56 vars in 1115 bytes} [Tue Sep 27 04:07:51 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:51.282758 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e14f45-8541-4539-bdfe-f62247c14b86 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:51.382102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e14f45-8541-4539-bdfe-f62247c14b86 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:51.416688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03483e5c-2169-4ad4-84a3-9276bee37ddf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:51.417523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03483e5c-2169-4ad4-84a3-9276bee37ddf None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:51.417964 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03483e5c-2169-4ad4-84a3-9276bee37ddf None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:51.418359 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03483e5c-2169-4ad4-84a3-9276bee37ddf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:51.429133 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03483e5c-2169-4ad4-84a3-9276bee37ddf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:51.430815 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03483e5c-2169-4ad4-84a3-9276bee37ddf None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:51.445628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-03483e5c-2169-4ad4-84a3-9276bee37ddf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:51.445628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:51.445628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:51.445628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:51.445628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:51.445628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:51.445628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:51.445628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:51.445628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:51.445628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:51.445628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:51.445628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:51.445628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:51.445628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:51.445628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:51.445628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:51.446756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:51.446756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:51.446756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:51.446756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:51.446756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:51.446756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:51.446756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:51.446756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:51.446756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:51.446756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:51.446756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:51.446756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:51.446756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:51.446756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:51.446756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:51.446756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:51.446756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:51.447844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:51.447844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:51.447844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:51.449479 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 447/888] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:51.455206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08b2df8-f938-4584-a116-d2eba252c122 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:51.542666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03483e5c-2169-4ad4-84a3-9276bee37ddf None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:51.560384 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:51.566278 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08b2df8-f938-4584-a116-d2eba252c122 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:51.579079 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:51.611475 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46695db9-b024-4b41-b3ca-cd93264160ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:51.612119 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46695db9-b024-4b41-b3ca-cd93264160ad None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:51.612516 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46695db9-b024-4b41-b3ca-cd93264160ad None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:51.612888 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46695db9-b024-4b41-b3ca-cd93264160ad None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:51.613817 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46695db9-b024-4b41-b3ca-cd93264160ad None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:51.614133 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46695db9-b024-4b41-b3ca-cd93264160ad None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:51.616238 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46695db9-b024-4b41-b3ca-cd93264160ad None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:51.633790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 442/889] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:51.637984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46695db9-b024-4b41-b3ca-cd93264160ad None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:51.671292 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03483e5c-2169-4ad4-84a3-9276bee37ddf None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:51.710783 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8307673-7f29-478b-93af-6748f43079df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:51.711464 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8307673-7f29-478b-93af-6748f43079df None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:51.711816 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8307673-7f29-478b-93af-6748f43079df None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:51.712172 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8307673-7f29-478b-93af-6748f43079df None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:51.712824 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8307673-7f29-478b-93af-6748f43079df None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:51.714284 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8307673-7f29-478b-93af-6748f43079df None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:51.724611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-c8307673-7f29-478b-93af-6748f43079df None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 27 04:07:51.724611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:51.724611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:07:51.724611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:07:51.724611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:07:51.724611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:07:51.724611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:07:51.724611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:51.724611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 27 04:07:51.724611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:51.724611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:07:51.724611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:51.724611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:51.724611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:51.724611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:51.724611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:51.724611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:51.725637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:51.725637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:51.725637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:51.725637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:51.725637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:51.725637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:51.725637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:07:51.725637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:07:51.725637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:07:51.725637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:07:51.725637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:51.725637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:51.725637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:51.725637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:51.725637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:51.725637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:51.731526 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:51.731526 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:51.731526 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:51.731526 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:51.731526 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:51.731526 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:51.731526 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:51.731526 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:51.731526 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:51.731526 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:51.731526 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:07:51.731526 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:51.731526 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 27 04:07:51.731526 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:51.731526 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 448/890] 10.222.0.26 () {56 vars in 1114 bytes} [Tue Sep 27 04:07:51 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:07:51.732862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8307673-7f29-478b-93af-6748f43079df None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:51.782972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46695db9-b024-4b41-b3ca-cd93264160ad None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:51.823702 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc33014-a230-4dab-9a2d-85da0e903dcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:51.824248 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc33014-a230-4dab-9a2d-85da0e903dcc None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:51.824525 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc33014-a230-4dab-9a2d-85da0e903dcc None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:51.824897 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc33014-a230-4dab-9a2d-85da0e903dcc None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:51.832336 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc33014-a230-4dab-9a2d-85da0e903dcc None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:51.833873 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc33014-a230-4dab-9a2d-85da0e903dcc None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:51.845577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-4dc33014-a230-4dab-9a2d-85da0e903dcc None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 27 04:07:51.845577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:51.845577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:51.845577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:51.845577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:51.845577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:51.845577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:51.845577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:51.845577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:51.845577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:51.845577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:51.845577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:51.845577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:51.845577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:51.845577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:51.845577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:51.846816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:51.846816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:51.846816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:51.846816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:51.846816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:51.846816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:51.846816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:51.846816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:51.846816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:51.846816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:51.846816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:51.846816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:51.846816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:51.846816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:51.846816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:51.846816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:51.846816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:51.846816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:51.847773 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:51.847773 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:51.847773 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:51.847773 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:51.847773 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:51.847773 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:51.847773 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:51.847773 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:51.847773 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 27 04:07:51.847773 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:51.851827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 443/891] 10.222.0.26 () {56 vars in 1117 bytes} [Tue Sep 27 04:07:51 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:07:51.853268 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc33014-a230-4dab-9a2d-85da0e903dcc None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:51.854055 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:51.862123 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8307673-7f29-478b-93af-6748f43079df None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:51.877235 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:51.904393 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d03a46-585f-44a3-a2af-4ea874cf2d5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:51.905126 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d03a46-585f-44a3-a2af-4ea874cf2d5d None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:51.905413 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d03a46-585f-44a3-a2af-4ea874cf2d5d None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:51.905685 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d03a46-585f-44a3-a2af-4ea874cf2d5d None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:51.906353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d03a46-585f-44a3-a2af-4ea874cf2d5d None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:51.906657 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d03a46-585f-44a3-a2af-4ea874cf2d5d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:51.908009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d03a46-585f-44a3-a2af-4ea874cf2d5d None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:51.918020 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 449/892] 10.222.0.26 () {56 vars in 1114 bytes} [Tue Sep 27 04:07:51 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:51.918020 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e65f3c-fec5-4d10-91a7-68049a883310 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:51.918020 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e65f3c-fec5-4d10-91a7-68049a883310 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:51.918020 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e65f3c-fec5-4d10-91a7-68049a883310 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:51.918020 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 450/893] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:51 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:51.919016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e65f3c-fec5-4d10-91a7-68049a883310 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:51.969526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc33014-a230-4dab-9a2d-85da0e903dcc None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:52.006185 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f324d445-1bf9-42a1-84f2-0f252da7435a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:52.006779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f324d445-1bf9-42a1-84f2-0f252da7435a None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:52.007051 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f324d445-1bf9-42a1-84f2-0f252da7435a None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:52.007315 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f324d445-1bf9-42a1-84f2-0f252da7435a None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:52.008030 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f324d445-1bf9-42a1-84f2-0f252da7435a None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:52.008327 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f324d445-1bf9-42a1-84f2-0f252da7435a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:52.009778 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f324d445-1bf9-42a1-84f2-0f252da7435a None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:52.027904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 444/894] 10.222.0.26 () {56 vars in 1149 bytes} [Tue Sep 27 04:07:51 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:07:52.029131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f867bc80-eaf4-4ab7-8307-f350137c435d None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:52.029412 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f867bc80-eaf4-4ab7-8307-f350137c435d None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:52.029682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f867bc80-eaf4-4ab7-8307-f350137c435d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:52.040094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e65f3c-fec5-4d10-91a7-68049a883310 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:52.047559 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-f867bc80-eaf4-4ab7-8307-f350137c435d None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:52.048742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-f867bc80-eaf4-4ab7-8307-f350137c435d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:52.076412 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a76fb2-a434-4bee-b400-c1ce84085040 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:52.076943 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a76fb2-a434-4bee-b400-c1ce84085040 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:52.077223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a76fb2-a434-4bee-b400-c1ce84085040 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:52.077493 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a76fb2-a434-4bee-b400-c1ce84085040 None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:52.078255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a76fb2-a434-4bee-b400-c1ce84085040 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:52.078458 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a76fb2-a434-4bee-b400-c1ce84085040 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:52.079806 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a76fb2-a434-4bee-b400-c1ce84085040 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:52.085050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 451/895] 10.222.0.26 () {56 vars in 1114 bytes} [Tue Sep 27 04:07:51 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:52.085840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a76fb2-a434-4bee-b400-c1ce84085040 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:52.111119 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:52.117119 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 445/896] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:52 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:07:52.118464 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4725743b-5816-43dd-a109-1fd4b10640e4 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:52.118749 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4725743b-5816-43dd-a109-1fd4b10640e4 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:52.119015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4725743b-5816-43dd-a109-1fd4b10640e4 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:52.120354 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 446/897] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:52.120951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4725743b-5816-43dd-a109-1fd4b10640e4 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:52.132224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:52.203303 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a76fb2-a434-4bee-b400-c1ce84085040 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:52.225182 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4725743b-5816-43dd-a109-1fd4b10640e4 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:52.239756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5180797-9a63-47ef-b402-b00db705d08e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:52.240356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5180797-9a63-47ef-b402-b00db705d08e None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:52.240638 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5180797-9a63-47ef-b402-b00db705d08e None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:52.240926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5180797-9a63-47ef-b402-b00db705d08e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:52.257706 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb26118-ecdc-4bf2-abda-232cec7d1ea6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:52.258318 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb26118-ecdc-4bf2-abda-232cec7d1ea6 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:52.258587 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb26118-ecdc-4bf2-abda-232cec7d1ea6 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:52.258859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb26118-ecdc-4bf2-abda-232cec7d1ea6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:52.259541 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb26118-ecdc-4bf2-abda-232cec7d1ea6 None admin] RBAC: Adding query filter params (interface=public, service_id=34ed09bed9184225b51c98ad35e0c9cc, region_id=RegionOne) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:52.259847 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb26118-ecdc-4bf2-abda-232cec7d1ea6 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:52.261196 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb26118-ecdc-4bf2-abda-232cec7d1ea6 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:52.271122 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 447/898] 10.222.0.26 () {56 vars in 1251 bytes} [Tue Sep 27 04:07:52 2022] GET /identity/v3/endpoints?service_id=34ed09bed9184225b51c98ad35e0c9cc&interface=public®ion_id=RegionOne => generated 194 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:52.271122 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936d5818-abca-4cbe-8be3-7346e760f045 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:52.271122 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936d5818-abca-4cbe-8be3-7346e760f045 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:52.271122 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936d5818-abca-4cbe-8be3-7346e760f045 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:52.287296 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-936d5818-abca-4cbe-8be3-7346e760f045 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:52.288450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-936d5818-abca-4cbe-8be3-7346e760f045 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:52.315982 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5180797-9a63-47ef-b402-b00db705d08e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:52.317478 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5180797-9a63-47ef-b402-b00db705d08e None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:52.342636 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:52.369766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 448/899] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:52 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:07:52.369766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936d5818-abca-4cbe-8be3-7346e760f045 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:52.372238 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:52.422372 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 452/900] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:07:52 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 04:07:52.422372 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5180797-9a63-47ef-b402-b00db705d08e None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:52.484045 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936d5818-abca-4cbe-8be3-7346e760f045 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:52.518898 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b678de4-6c50-460d-8381-1715f33ff516 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:52.519494 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b678de4-6c50-460d-8381-1715f33ff516 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:52.519767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b678de4-6c50-460d-8381-1715f33ff516 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:52.520037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b678de4-6c50-460d-8381-1715f33ff516 None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:52.521242 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b678de4-6c50-460d-8381-1715f33ff516 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:52.522632 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b678de4-6c50-460d-8381-1715f33ff516 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:52.537778 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 449/901] 10.222.0.26 () {56 vars in 1087 bytes} [Tue Sep 27 04:07:52 2022] GET /identity/v3/services => generated 1268 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:07:52.537778 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b678de4-6c50-460d-8381-1715f33ff516 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:52.541116 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5180797-9a63-47ef-b402-b00db705d08e None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:52.568360 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:52.582713 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae848ad7-97cd-40ea-adb3-ac1f017c7608 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:52.583965 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae848ad7-97cd-40ea-adb3-ac1f017c7608 None admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:52.584997 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae848ad7-97cd-40ea-adb3-ac1f017c7608 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:52.584997 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae848ad7-97cd-40ea-adb3-ac1f017c7608 None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:52.585358 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae848ad7-97cd-40ea-adb3-ac1f017c7608 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:52.586798 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae848ad7-97cd-40ea-adb3-ac1f017c7608 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:52.604127 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 453/902] 10.222.0.26 () {60 vars in 1141 bytes} [Tue Sep 27 04:07:52 2022] POST /identity/v3/services => generated 241 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:07:52.617901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:52.637153 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae848ad7-97cd-40ea-adb3-ac1f017c7608 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:52.677089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b678de4-6c50-460d-8381-1715f33ff516 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:52.737223 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265a6097-5b8d-45cb-bfcf-81aa1fe6376e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:52.738704 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265a6097-5b8d-45cb-bfcf-81aa1fe6376e None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:52.739186 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265a6097-5b8d-45cb-bfcf-81aa1fe6376e None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:52.740040 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265a6097-5b8d-45cb-bfcf-81aa1fe6376e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:52.754793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265a6097-5b8d-45cb-bfcf-81aa1fe6376e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:52.757218 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265a6097-5b8d-45cb-bfcf-81aa1fe6376e None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:52.769363 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae848ad7-97cd-40ea-adb3-ac1f017c7608 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:52.776684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-265a6097-5b8d-45cb-bfcf-81aa1fe6376e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:52.776684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:52.776684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:52.776684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:52.776684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:52.776684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:52.776684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:52.776684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:52.776684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:52.776684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:52.776684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:52.776684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:52.776684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:52.776684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:52.776684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:52.776684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:52.777684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:52.777684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:52.777684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:52.777684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:52.777684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:52.777684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:52.777684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:52.777684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:52.777684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:52.777684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:52.777684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:52.777684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:52.777684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:52.777684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:52.777684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:52.777684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:52.777684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:52.778631 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:52.778631 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:52.778631 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:52.779448 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 450/903] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:52.785136 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265a6097-5b8d-45cb-bfcf-81aa1fe6376e None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:52.824702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c02279-f183-4205-a010-bb2412db78df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:52.825329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c02279-f183-4205-a010-bb2412db78df None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:52.826243 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c02279-f183-4205-a010-bb2412db78df None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:52.826243 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c02279-f183-4205-a010-bb2412db78df None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:52.826508 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c02279-f183-4205-a010-bb2412db78df None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:52.827873 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c02279-f183-4205-a010-bb2412db78df None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:52.837488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-04c02279-f183-4205-a010-bb2412db78df None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 27 04:07:52.837488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:52.837488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:07:52.837488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:07:52.837488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:07:52.837488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:07:52.837488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:07:52.837488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:52.837488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 27 04:07:52.837488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:52.837488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:07:52.837488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:52.837488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:52.837488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:52.837488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:52.837488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:52.838624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:52.838624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:52.838624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:52.838624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:52.838624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:52.838624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:52.838624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:52.838624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:07:52.838624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:07:52.838624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:07:52.838624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:07:52.838624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:52.838624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:52.838624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:52.838624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:52.838624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:52.838624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:52.839500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:52.839500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:52.839500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:52.839500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:52.839500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:52.839500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:52.839500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:52.839500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:52.839500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:52.839500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:52.839500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:07:52.839500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:52.839500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 27 04:07:52.839500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:52.843432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 454/904] 10.222.0.26 () {56 vars in 1132 bytes} [Tue Sep 27 04:07:52 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:07:52.846413 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c02279-f183-4205-a010-bb2412db78df None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:52.896336 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:52.912505 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265a6097-5b8d-45cb-bfcf-81aa1fe6376e None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:52.931459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:52.958072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195a874e-3ad2-4e86-bbab-ae52f173dc89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:52.958677 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195a874e-3ad2-4e86-bbab-ae52f173dc89 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:52.958996 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195a874e-3ad2-4e86-bbab-ae52f173dc89 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:52.959270 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195a874e-3ad2-4e86-bbab-ae52f173dc89 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:52.960182 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195a874e-3ad2-4e86-bbab-ae52f173dc89 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:52.960487 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195a874e-3ad2-4e86-bbab-ae52f173dc89 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:52.962598 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195a874e-3ad2-4e86-bbab-ae52f173dc89 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:52.967481 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c02279-f183-4205-a010-bb2412db78df None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:52.985503 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 451/905] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:07:52.986595 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195a874e-3ad2-4e86-bbab-ae52f173dc89 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:53.014360 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fbe4580-f219-43ff-98c3-898a0819ca6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:53.015340 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbe4580-f219-43ff-98c3-898a0819ca6f None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:53.015340 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbe4580-f219-43ff-98c3-898a0819ca6f None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:53.015595 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbe4580-f219-43ff-98c3-898a0819ca6f None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:53.016238 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fbe4580-f219-43ff-98c3-898a0819ca6f None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:53.016540 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fbe4580-f219-43ff-98c3-898a0819ca6f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:53.017979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fbe4580-f219-43ff-98c3-898a0819ca6f None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:53.024180 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 455/906] 10.222.0.26 () {56 vars in 1126 bytes} [Tue Sep 27 04:07:52 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:53.029379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fbe4580-f219-43ff-98c3-898a0819ca6f None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:53.124659 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195a874e-3ad2-4e86-bbab-ae52f173dc89 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:53.160381 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fbe4580-f219-43ff-98c3-898a0819ca6f None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:53.169814 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d4f6ae-4487-4f64-9835-3cd98c1435dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:53.170453 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d4f6ae-4487-4f64-9835-3cd98c1435dc None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:53.170730 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d4f6ae-4487-4f64-9835-3cd98c1435dc None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:53.171001 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d4f6ae-4487-4f64-9835-3cd98c1435dc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:53.174532 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:53.183515 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d4f6ae-4487-4f64-9835-3cd98c1435dc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:53.184964 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d4f6ae-4487-4f64-9835-3cd98c1435dc None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:53.200760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-07d4f6ae-4487-4f64-9835-3cd98c1435dc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:53.200760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:53.200760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:53.200760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:53.200760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:53.200760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:53.200760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:53.200760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:53.200760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:53.200760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:53.200760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:53.200760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:53.200760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:53.200760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:53.200760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:53.201763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:53.201763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:53.201763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:53.201763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:53.201763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:53.201763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:53.201763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:53.201763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:53.201763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:53.201763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:53.201763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:53.201763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:53.201763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:53.201763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:53.201763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:53.201763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:53.201763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:53.202720 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:53.202720 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:53.202720 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:53.202720 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:53.202720 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:53.202720 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:53.202720 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:53.202720 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:53.205324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f65153f-0be2-43bc-976a-e6c2db825eec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:53.206261 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f65153f-0be2-43bc-976a-e6c2db825eec None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:53.206662 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 452/907] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:53.206971 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f65153f-0be2-43bc-976a-e6c2db825eec None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:53.207340 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f65153f-0be2-43bc-976a-e6c2db825eec None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:53.208103 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f65153f-0be2-43bc-976a-e6c2db825eec None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:53.208495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f65153f-0be2-43bc-976a-e6c2db825eec None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:53.208853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d4f6ae-4487-4f64-9835-3cd98c1435dc None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:53.210379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f65153f-0be2-43bc-976a-e6c2db825eec None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:53.220991 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 456/908] 10.222.0.26 () {56 vars in 1126 bytes} [Tue Sep 27 04:07:53 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:07:53.222830 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f65153f-0be2-43bc-976a-e6c2db825eec None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:53.249901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:53.318871 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d4f6ae-4487-4f64-9835-3cd98c1435dc None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:53.344196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f65153f-0be2-43bc-976a-e6c2db825eec None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:53.355803 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26d59e1-787b-45ac-8477-45106a839fc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:53.356501 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26d59e1-787b-45ac-8477-45106a839fc0 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:53.357066 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26d59e1-787b-45ac-8477-45106a839fc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:53.357066 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26d59e1-787b-45ac-8477-45106a839fc0 None admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:53.357891 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26d59e1-787b-45ac-8477-45106a839fc0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:53.358259 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26d59e1-787b-45ac-8477-45106a839fc0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:53.359759 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26d59e1-787b-45ac-8477-45106a839fc0 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:53.378169 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 453/909] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:53.379198 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26d59e1-787b-45ac-8477-45106a839fc0 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:53.386012 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebeecc6a-a4d3-4aa4-8eb2-8b579b336900 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:53.386962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebeecc6a-a4d3-4aa4-8eb2-8b579b336900 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:53.387420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebeecc6a-a4d3-4aa4-8eb2-8b579b336900 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:53.387866 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebeecc6a-a4d3-4aa4-8eb2-8b579b336900 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:53.388748 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebeecc6a-a4d3-4aa4-8eb2-8b579b336900 None admin] RBAC: Adding query filter params (interface=public, service_id=2fafcd56c9114bdda23cfc724cca7d88, region_id=RegionOne) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:53.389656 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebeecc6a-a4d3-4aa4-8eb2-8b579b336900 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:53.391581 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebeecc6a-a4d3-4aa4-8eb2-8b579b336900 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:53.402660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 457/910] 10.222.0.26 () {56 vars in 1251 bytes} [Tue Sep 27 04:07:53 2022] GET /identity/v3/endpoints?service_id=2fafcd56c9114bdda23cfc724cca7d88&interface=public®ion_id=RegionOne => generated 194 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:07:53.404184 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebeecc6a-a4d3-4aa4-8eb2-8b579b336900 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:53.453525 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:53.502305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:53.513114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26d59e1-787b-45ac-8477-45106a839fc0 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:53.530827 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebeecc6a-a4d3-4aa4-8eb2-8b579b336900 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:53.556714 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e21d5fa-2d40-4196-ada5-76b212d63221 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:53.557650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e21d5fa-2d40-4196-ada5-76b212d63221 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:53.558167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e21d5fa-2d40-4196-ada5-76b212d63221 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:53.558617 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e21d5fa-2d40-4196-ada5-76b212d63221 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:53.559539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e21d5fa-2d40-4196-ada5-76b212d63221 None admin] RBAC: Adding query filter params (role.id=d25f9751bd8d4cb3963a382887a028ef, scope.project.id=c830a3bfece6456486f4b094076b5364, user.id=5a2fcc8c9ac44dceadcc30d163bbfa0e) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:53.560045 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e21d5fa-2d40-4196-ada5-76b212d63221 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:53.561707 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e21d5fa-2d40-4196-ada5-76b212d63221 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:53.573767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a508275a-2b7c-432b-ab78-7ab4dfe47682 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:53.573767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a508275a-2b7c-432b-ab78-7ab4dfe47682 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:53.573767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a508275a-2b7c-432b-ab78-7ab4dfe47682 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:53.573767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a508275a-2b7c-432b-ab78-7ab4dfe47682 None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:53.574197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a508275a-2b7c-432b-ab78-7ab4dfe47682 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:53.574197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a508275a-2b7c-432b-ab78-7ab4dfe47682 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:53.574988 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 454/911] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:07:53 2022] GET /identity/v3/role_assignments?user.id=5a2fcc8c9ac44dceadcc30d163bbfa0e&scope.project.id=c830a3bfece6456486f4b094076b5364&role.id=d25f9751bd8d4cb3963a382887a028ef => generated 259 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:53.581463 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 458/912] 10.222.0.26 () {56 vars in 1087 bytes} [Tue Sep 27 04:07:53 2022] GET /identity/v3/services => generated 1497 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:07:54.779851 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d458d76-c155-4175-96b3-2ef093d67cb5 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:54.780428 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d458d76-c155-4175-96b3-2ef093d67cb5 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:54.780428 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d458d76-c155-4175-96b3-2ef093d67cb5 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:54.781968 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 455/913] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:54.787883 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3285e9-85e2-41f5-b9ea-f331ad53bcf4 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:54.788783 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3285e9-85e2-41f5-b9ea-f331ad53bcf4 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:54.789270 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3285e9-85e2-41f5-b9ea-f331ad53bcf4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:54.808128 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-ec3285e9-85e2-41f5-b9ea-f331ad53bcf4 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:54.809644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ec3285e9-85e2-41f5-b9ea-f331ad53bcf4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:54.872714 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e3d707-b6f5-42bb-b4d5-589a98fe212c None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:54.872991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e3d707-b6f5-42bb-b4d5-589a98fe212c None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:54.873264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e3d707-b6f5-42bb-b4d5-589a98fe212c None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:54.874809 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 456/914] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:54.880823 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76af0d9-d12d-43c6-945c-dde876e796c9 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:54.881098 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76af0d9-d12d-43c6-945c-dde876e796c9 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:54.881368 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76af0d9-d12d-43c6-945c-dde876e796c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:54.882338 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 459/915] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:54 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 04:07:54.903940 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-c76af0d9-d12d-43c6-945c-dde876e796c9 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:54.905113 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c76af0d9-d12d-43c6-945c-dde876e796c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:54.972142 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 457/916] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:54 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 04:07:55.153696 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3285e9-85e2-41f5-b9ea-f331ad53bcf4 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:55.184278 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:55.243696 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76af0d9-d12d-43c6-945c-dde876e796c9 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:55.260151 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3285e9-85e2-41f5-b9ea-f331ad53bcf4 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:55.299573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed5b896-4881-4440-a249-9884faf63788 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:55.300208 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed5b896-4881-4440-a249-9884faf63788 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:55.300487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed5b896-4881-4440-a249-9884faf63788 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:55.300758 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed5b896-4881-4440-a249-9884faf63788 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:55.301345 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed5b896-4881-4440-a249-9884faf63788 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:55.303256 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed5b896-4881-4440-a249-9884faf63788 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:55.313368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-fed5b896-4881-4440-a249-9884faf63788 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 27 04:07:55.313368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:55.313368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:07:55.313368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:07:55.313368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:07:55.313368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:07:55.313368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:07:55.313368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:55.313368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 27 04:07:55.313368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:55.313368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:07:55.313368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:55.313368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:55.313368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:55.313368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:55.313368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:55.313368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:55.314660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:55.314660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:55.314660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:55.314660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:55.314660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:55.314660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:55.314660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:07:55.314660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:07:55.314660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:07:55.314660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:07:55.314660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:55.314660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:55.314660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:55.314660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:55.314660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:55.314660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:55.315842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:55.315842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:55.315842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:55.315842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:55.315842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:55.315842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:55.315842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:55.315842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:55.315842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:55.315842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:55.315842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:07:55.315842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:55.315842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 27 04:07:55.315842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:55.315842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 460/917] 10.222.0.26 () {56 vars in 1114 bytes} [Tue Sep 27 04:07:55 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:55.322669 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed5b896-4881-4440-a249-9884faf63788 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:55.344727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:55.357570 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76af0d9-d12d-43c6-945c-dde876e796c9 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:55.396522 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87205c89-120a-4efd-bee6-1038d19d93bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:55.397369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87205c89-120a-4efd-bee6-1038d19d93bb None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:55.397882 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87205c89-120a-4efd-bee6-1038d19d93bb None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:55.398246 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87205c89-120a-4efd-bee6-1038d19d93bb None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:55.399417 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87205c89-120a-4efd-bee6-1038d19d93bb None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:55.406105 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87205c89-120a-4efd-bee6-1038d19d93bb None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:55.425894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-87205c89-120a-4efd-bee6-1038d19d93bb None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 04:07:55.425894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:55.425894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:07:55.425894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:07:55.425894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:07:55.425894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:07:55.425894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:07:55.425894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:55.425894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 04:07:55.425894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:55.425894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:07:55.425894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:55.425894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:55.425894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:55.425894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:55.425894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:55.425894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:55.427529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:55.427529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:55.427529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:55.427529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:55.427529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:55.427529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:55.427529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:07:55.427529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:07:55.427529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:07:55.427529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:07:55.427529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:55.427529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:55.427529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:55.427529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:55.427529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:55.427529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:55.428399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:55.428399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:55.428399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:55.428399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:55.428399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:55.428399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:55.428399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:55.428399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:55.428399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:55.428399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:55.428399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:07:55.428399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:55.428399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 04:07:55.428399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:55.428399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 458/918] 10.222.0.26 () {56 vars in 1126 bytes} [Tue Sep 27 04:07:55 2022] GET /identity/v3/services/object-store => generated 93 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:55.433273 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87205c89-120a-4efd-bee6-1038d19d93bb None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:55.452720 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed5b896-4881-4440-a249-9884faf63788 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:55.466155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:55.501680 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf85e67-c71f-470d-b5eb-52ed17454f20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:55.502586 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf85e67-c71f-470d-b5eb-52ed17454f20 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:55.503021 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf85e67-c71f-470d-b5eb-52ed17454f20 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:55.503431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf85e67-c71f-470d-b5eb-52ed17454f20 None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:55.504446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf85e67-c71f-470d-b5eb-52ed17454f20 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:55.505245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf85e67-c71f-470d-b5eb-52ed17454f20 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:55.507207 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf85e67-c71f-470d-b5eb-52ed17454f20 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:55.515436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 461/919] 10.222.0.26 () {56 vars in 1114 bytes} [Tue Sep 27 04:07:55 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:55.520733 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf85e67-c71f-470d-b5eb-52ed17454f20 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:55.553018 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87205c89-120a-4efd-bee6-1038d19d93bb None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:55.599996 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a62129-332e-46ce-99d3-e699c774ea8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:55.601418 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a62129-332e-46ce-99d3-e699c774ea8b None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:55.601890 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a62129-332e-46ce-99d3-e699c774ea8b None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:55.602302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a62129-332e-46ce-99d3-e699c774ea8b None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:55.603494 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a62129-332e-46ce-99d3-e699c774ea8b None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:55.603964 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a62129-332e-46ce-99d3-e699c774ea8b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:55.606194 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a62129-332e-46ce-99d3-e699c774ea8b None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:55.612999 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 459/920] 10.222.0.26 () {56 vars in 1122 bytes} [Tue Sep 27 04:07:55 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:07:55.619306 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a62129-332e-46ce-99d3-e699c774ea8b None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:55.632306 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf85e67-c71f-470d-b5eb-52ed17454f20 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:55.638694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:55.669979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ab5d54-e22d-41da-bc3b-f82c4552e8cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:55.670677 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ab5d54-e22d-41da-bc3b-f82c4552e8cf None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:55.671047 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ab5d54-e22d-41da-bc3b-f82c4552e8cf None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:55.671440 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ab5d54-e22d-41da-bc3b-f82c4552e8cf None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:55.672236 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ab5d54-e22d-41da-bc3b-f82c4552e8cf None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:55.672636 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ab5d54-e22d-41da-bc3b-f82c4552e8cf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:55.674104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ab5d54-e22d-41da-bc3b-f82c4552e8cf None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:55.681948 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 462/921] 10.222.0.26 () {56 vars in 1114 bytes} [Tue Sep 27 04:07:55 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:55.688199 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ab5d54-e22d-41da-bc3b-f82c4552e8cf None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:55.719164 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:55.767196 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a62129-332e-46ce-99d3-e699c774ea8b None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:55.804180 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ab5d54-e22d-41da-bc3b-f82c4552e8cf None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:55.806796 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e345d1e-3ec3-42ba-a8bd-d90385863079 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:55.807891 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e345d1e-3ec3-42ba-a8bd-d90385863079 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:55.808360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e345d1e-3ec3-42ba-a8bd-d90385863079 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:55.808837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e345d1e-3ec3-42ba-a8bd-d90385863079 None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:55.809671 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e345d1e-3ec3-42ba-a8bd-d90385863079 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:55.810255 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e345d1e-3ec3-42ba-a8bd-d90385863079 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:55.811918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e345d1e-3ec3-42ba-a8bd-d90385863079 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:55.817770 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 460/922] 10.222.0.26 () {56 vars in 1122 bytes} [Tue Sep 27 04:07:55 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:55.823167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e345d1e-3ec3-42ba-a8bd-d90385863079 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:55.840254 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14075fc9-9b8a-4adc-a8f3-ef39fe45726c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:55.840863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14075fc9-9b8a-4adc-a8f3-ef39fe45726c None admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:55.841144 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14075fc9-9b8a-4adc-a8f3-ef39fe45726c None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:55.841416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14075fc9-9b8a-4adc-a8f3-ef39fe45726c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:55.842110 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14075fc9-9b8a-4adc-a8f3-ef39fe45726c None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:55.843465 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14075fc9-9b8a-4adc-a8f3-ef39fe45726c None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:55.862945 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 463/923] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:07:55 2022] POST /identity/v3/endpoints => generated 346 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:07:55.864133 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68559309-2dcb-4a32-8bdc-ed73e3a18170 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:55.864452 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68559309-2dcb-4a32-8bdc-ed73e3a18170 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:55.864728 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68559309-2dcb-4a32-8bdc-ed73e3a18170 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:55.866116 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 464/924] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:55 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:55.866999 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eaad80e-122a-43f6-a488-35699990b6b3 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:55.867373 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eaad80e-122a-43f6-a488-35699990b6b3 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:55.867649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eaad80e-122a-43f6-a488-35699990b6b3 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:55.872924 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 465/925] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:55.872924 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfebd6f-d96c-4813-8845-a2b2931ebaf2 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:55.872924 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfebd6f-d96c-4813-8845-a2b2931ebaf2 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:55.872924 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfebd6f-d96c-4813-8845-a2b2931ebaf2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:55.891760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-0bfebd6f-d96c-4813-8845-a2b2931ebaf2 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:55.892984 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0bfebd6f-d96c-4813-8845-a2b2931ebaf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:55.919712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:55.940737 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e345d1e-3ec3-42ba-a8bd-d90385863079 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:55.985817 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 466/926] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:55 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 04:07:55.985817 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ef9b23-8c93-4ab0-8de9-d9c928ada554 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:55.985817 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ef9b23-8c93-4ab0-8de9-d9c928ada554 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:55.985817 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ef9b23-8c93-4ab0-8de9-d9c928ada554 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:55.991295 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3aaaddc-ec5c-4b8f-895c-b99a02416a6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:55.992213 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3aaaddc-ec5c-4b8f-895c-b99a02416a6c None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:55.992693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3aaaddc-ec5c-4b8f-895c-b99a02416a6c None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:55.993153 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3aaaddc-ec5c-4b8f-895c-b99a02416a6c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:55.994086 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3aaaddc-ec5c-4b8f-895c-b99a02416a6c None admin] RBAC: Adding query filter params (interface=public, service_id=55a0fac13c71463bad022337ce1dcd45, region_id=RegionOne) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:55.994609 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3aaaddc-ec5c-4b8f-895c-b99a02416a6c None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:55.996005 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3aaaddc-ec5c-4b8f-895c-b99a02416a6c None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:56.000758 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:56.003747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 461/927] 10.222.0.26 () {56 vars in 1251 bytes} [Tue Sep 27 04:07:55 2022] GET /identity/v3/endpoints?service_id=55a0fac13c71463bad022337ce1dcd45&interface=public®ion_id=RegionOne => generated 194 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:56.008669 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-b2ef9b23-8c93-4ab0-8de9-d9c928ada554 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:56.010571 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-b2ef9b23-8c93-4ab0-8de9-d9c928ada554 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:56.010906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3aaaddc-ec5c-4b8f-895c-b99a02416a6c None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:56.100060 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 467/928] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:55 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:07:56.132348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3aaaddc-ec5c-4b8f-895c-b99a02416a6c None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:56.169329 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0524cefa-6925-4f8d-a483-f2da896c55de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:56.169958 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0524cefa-6925-4f8d-a483-f2da896c55de None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:56.170233 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0524cefa-6925-4f8d-a483-f2da896c55de None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:56.170499 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0524cefa-6925-4f8d-a483-f2da896c55de None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:56.171094 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0524cefa-6925-4f8d-a483-f2da896c55de None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:56.172440 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0524cefa-6925-4f8d-a483-f2da896c55de None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:56.180359 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 462/929] 10.222.0.26 () {56 vars in 1087 bytes} [Tue Sep 27 04:07:56 2022] GET /identity/v3/services => generated 1497 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:07:56.289378 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ef9b23-8c93-4ab0-8de9-d9c928ada554 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:56.385357 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0524cefa-6925-4f8d-a483-f2da896c55de None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:56.389190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:56.400927 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ef9b23-8c93-4ab0-8de9-d9c928ada554 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:56.436831 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9811cb18-7ea2-4a50-b6bb-7e3573556ead None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:56.437464 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9811cb18-7ea2-4a50-b6bb-7e3573556ead None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:56.437737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9811cb18-7ea2-4a50-b6bb-7e3573556ead None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:56.438030 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9811cb18-7ea2-4a50-b6bb-7e3573556ead None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:56.438637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9811cb18-7ea2-4a50-b6bb-7e3573556ead None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:56.440073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9811cb18-7ea2-4a50-b6bb-7e3573556ead None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:56.443413 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:56.450659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-9811cb18-7ea2-4a50-b6bb-7e3573556ead None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 27 04:07:56.450659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:56.450659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:07:56.450659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:07:56.450659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:07:56.450659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:07:56.450659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:07:56.450659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:56.450659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 27 04:07:56.450659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:56.450659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:07:56.450659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:56.450659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:56.450659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:56.450659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:56.450659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:56.451840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:56.451840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:56.451840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:56.451840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:56.451840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:56.451840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:56.451840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:56.451840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:07:56.451840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:07:56.451840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:07:56.451840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:07:56.451840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:56.451840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:56.451840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:56.451840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:56.451840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:56.451840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:56.452950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:56.452950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:56.452950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:56.452950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:56.452950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:56.452950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:56.452950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:56.452950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:56.452950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:56.452950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:56.452950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:07:56.452950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:56.452950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 27 04:07:56.452950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:56.455765 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 468/930] 10.222.0.26 () {56 vars in 1132 bytes} [Tue Sep 27 04:07:56 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:56.462132 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9811cb18-7ea2-4a50-b6bb-7e3573556ead None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:56.507112 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0524cefa-6925-4f8d-a483-f2da896c55de None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:56.545209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2339eb06-7b16-4171-aa98-ffd13244a230 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:56.545966 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2339eb06-7b16-4171-aa98-ffd13244a230 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:56.546249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2339eb06-7b16-4171-aa98-ffd13244a230 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:56.546523 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2339eb06-7b16-4171-aa98-ffd13244a230 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:56.560549 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2339eb06-7b16-4171-aa98-ffd13244a230 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:07:56.567815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-2339eb06-7b16-4171-aa98-ffd13244a230 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 27 04:07:56.567815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:56.567815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:56.567815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:56.567815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:56.567815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:56.567815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:56.567815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:56.567815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:56.567815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:56.567815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:56.567815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:56.567815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:07:56.567815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:07:56.567815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:07:56.567815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:07:56.569102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:07:56.569102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:07:56.569102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:07:56.569102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:07:56.569102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:56.569102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:56.569102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:56.569102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:56.569102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:56.569102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:56.569102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:56.569102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:56.569102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:56.569102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:56.569102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:56.569102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:56.569990 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:56.569990 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:56.569990 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:56.569990 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:56.569990 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:07:56.569990 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:07:56.569990 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:07:56.569990 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:07:56.569990 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:07:56.569990 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:07:56.569990 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 27 04:07:56.569990 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:56.569990 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 463/931] 10.222.0.26 () {56 vars in 1120 bytes} [Tue Sep 27 04:07:56 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:56.575050 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2339eb06-7b16-4171-aa98-ffd13244a230 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:56.593759 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9811cb18-7ea2-4a50-b6bb-7e3573556ead None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:56.629429 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d8e5a5-5289-4a39-bf03-0855b7a7e9f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:56.630725 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d8e5a5-5289-4a39-bf03-0855b7a7e9f0 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:56.631193 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d8e5a5-5289-4a39-bf03-0855b7a7e9f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:56.631668 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d8e5a5-5289-4a39-bf03-0855b7a7e9f0 None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:56.632507 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d8e5a5-5289-4a39-bf03-0855b7a7e9f0 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:56.632997 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d8e5a5-5289-4a39-bf03-0855b7a7e9f0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:56.634672 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d8e5a5-5289-4a39-bf03-0855b7a7e9f0 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:56.640126 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 469/932] 10.222.0.26 () {56 vars in 1126 bytes} [Tue Sep 27 04:07:56 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:07:56.646273 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d8e5a5-5289-4a39-bf03-0855b7a7e9f0 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:56.648265 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:56.691957 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2339eb06-7b16-4171-aa98-ffd13244a230 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:56.710174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:56.739399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbfbd8b-21e1-4521-9b33-663e08512ea7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:56.740141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbfbd8b-21e1-4521-9b33-663e08512ea7 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:56.740538 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbfbd8b-21e1-4521-9b33-663e08512ea7 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:56.740818 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbfbd8b-21e1-4521-9b33-663e08512ea7 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:56.741481 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efbfbd8b-21e1-4521-9b33-663e08512ea7 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:56.741793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efbfbd8b-21e1-4521-9b33-663e08512ea7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:56.743631 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efbfbd8b-21e1-4521-9b33-663e08512ea7 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:56.751785 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d8e5a5-5289-4a39-bf03-0855b7a7e9f0 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:56.757786 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 464/933] 10.222.0.26 () {56 vars in 1115 bytes} [Tue Sep 27 04:07:56 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:07:56.760409 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbfbd8b-21e1-4521-9b33-663e08512ea7 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:56.794181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f1202f-4667-44d9-8694-097e35fdb074 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:56.794921 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f1202f-4667-44d9-8694-097e35fdb074 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:56.795352 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f1202f-4667-44d9-8694-097e35fdb074 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:56.795659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f1202f-4667-44d9-8694-097e35fdb074 None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:56.796379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f1202f-4667-44d9-8694-097e35fdb074 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:56.796771 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f1202f-4667-44d9-8694-097e35fdb074 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:56.798254 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f1202f-4667-44d9-8694-097e35fdb074 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:56.808011 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 470/934] 10.222.0.26 () {56 vars in 1126 bytes} [Tue Sep 27 04:07:56 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:07:56.810665 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f1202f-4667-44d9-8694-097e35fdb074 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:56.884199 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbfbd8b-21e1-4521-9b33-663e08512ea7 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:56.914912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:56.925428 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ee5570-e8a7-4501-9bf0-0280cd2478b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:56.926083 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ee5570-e8a7-4501-9bf0-0280cd2478b2 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:56.926358 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ee5570-e8a7-4501-9bf0-0280cd2478b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:56.926627 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ee5570-e8a7-4501-9bf0-0280cd2478b2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:56.937832 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ee5570-e8a7-4501-9bf0-0280cd2478b2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:56.939263 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ee5570-e8a7-4501-9bf0-0280cd2478b2 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:56.953871 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f1202f-4667-44d9-8694-097e35fdb074 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:56.954330 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-f0ee5570-e8a7-4501-9bf0-0280cd2478b2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:56.954330 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:56.954330 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:56.954330 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:56.954330 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:56.954330 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:56.954330 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:56.954330 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:56.954330 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:56.954330 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:56.954330 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:56.954330 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:56.954330 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:56.954330 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:56.954330 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:56.954330 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:56.955361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:56.955361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:56.955361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:56.955361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:56.955361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:56.955361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:56.955361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:56.955361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:56.955361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:56.955361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:56.955361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:56.955361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:56.955361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:56.955361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:56.955361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:56.955361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:56.955361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:56.956264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:56.956264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:56.956264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:56.956264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 465/935] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:07:56.961504 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ee5570-e8a7-4501-9bf0-0280cd2478b2 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:56.993508 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:57.004043 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca8ab0e-2b33-4479-9a7c-c223dcbf7ff0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:57.005906 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca8ab0e-2b33-4479-9a7c-c223dcbf7ff0 None admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:57.007188 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca8ab0e-2b33-4479-9a7c-c223dcbf7ff0 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:57.007613 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca8ab0e-2b33-4479-9a7c-c223dcbf7ff0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:57.008447 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca8ab0e-2b33-4479-9a7c-c223dcbf7ff0 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:57.010038 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca8ab0e-2b33-4479-9a7c-c223dcbf7ff0 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:57.039263 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 471/936] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:07:56 2022] POST /identity/v3/endpoints => generated 347 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:07:57.082789 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ee5570-e8a7-4501-9bf0-0280cd2478b2 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:57.127722 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33eeadda-5b44-4297-9de5-143253068cad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:57.128672 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33eeadda-5b44-4297-9de5-143253068cad None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:57.129143 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33eeadda-5b44-4297-9de5-143253068cad None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:57.130418 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33eeadda-5b44-4297-9de5-143253068cad None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:57.131235 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33eeadda-5b44-4297-9de5-143253068cad None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:57.131729 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33eeadda-5b44-4297-9de5-143253068cad None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:57.133317 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33eeadda-5b44-4297-9de5-143253068cad None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:57.151067 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 466/937] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:57.156852 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca8ab0e-2b33-4479-9a7c-c223dcbf7ff0 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:57.268469 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca8ab0e-2b33-4479-9a7c-c223dcbf7ff0 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:57.303628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65194b63-676d-4e98-98e3-e0aa73250114 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:57.304250 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65194b63-676d-4e98-98e3-e0aa73250114 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:57.304526 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65194b63-676d-4e98-98e3-e0aa73250114 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:57.304798 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65194b63-676d-4e98-98e3-e0aa73250114 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:57.312452 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65194b63-676d-4e98-98e3-e0aa73250114 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:57.313888 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65194b63-676d-4e98-98e3-e0aa73250114 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:57.325667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-65194b63-676d-4e98-98e3-e0aa73250114 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 27 04:07:57.325667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:57.325667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:57.325667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:57.325667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:57.325667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:57.325667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:57.325667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:57.325667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:57.325667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:57.325667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:57.325667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:57.325667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:07:57.325667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:07:57.325667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:07:57.325667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:07:57.326660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:57.326660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:57.326660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:07:57.326660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:57.326660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:07:57.326660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:07:57.326660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:57.326660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:57.326660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:57.326660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:57.326660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:57.326660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:57.326660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:57.326660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:57.326660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:57.326660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:57.326660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:57.326660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:57.327614 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:57.327614 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:57.327614 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:07:57.327614 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:07:57.327614 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:07:57.327614 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:07:57.327614 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:07:57.327614 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:07:57.327614 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 27 04:07:57.327614 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:57.327614 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 472/938] 10.222.0.26 () {56 vars in 1117 bytes} [Tue Sep 27 04:07:57 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:57.332418 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33eeadda-5b44-4297-9de5-143253068cad None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:57.366328 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:57.440023 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33eeadda-5b44-4297-9de5-143253068cad None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:57.475152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8e59a0-2201-43b3-ae77-abf23dcf6c2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:57.476097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8e59a0-2201-43b3-ae77-abf23dcf6c2c None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:57.476570 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8e59a0-2201-43b3-ae77-abf23dcf6c2c None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:57.477054 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8e59a0-2201-43b3-ae77-abf23dcf6c2c None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:57.478024 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8e59a0-2201-43b3-ae77-abf23dcf6c2c None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:57.478525 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8e59a0-2201-43b3-ae77-abf23dcf6c2c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:57.480185 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8e59a0-2201-43b3-ae77-abf23dcf6c2c None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:57.499040 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 467/939] 10.222.0.26 () {56 vars in 1149 bytes} [Tue Sep 27 04:07:57 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:57.504312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65194b63-676d-4e98-98e3-e0aa73250114 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:57.553104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:57.611026 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65194b63-676d-4e98-98e3-e0aa73250114 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:57.645857 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0066dbf6-40ed-4cc5-aab7-457ceb4aabb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:57.646825 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0066dbf6-40ed-4cc5-aab7-457ceb4aabb6 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:57.647321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0066dbf6-40ed-4cc5-aab7-457ceb4aabb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:57.647791 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0066dbf6-40ed-4cc5-aab7-457ceb4aabb6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:57.663765 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0066dbf6-40ed-4cc5-aab7-457ceb4aabb6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:57.665206 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0066dbf6-40ed-4cc5-aab7-457ceb4aabb6 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:57.679909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-0066dbf6-40ed-4cc5-aab7-457ceb4aabb6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:57.679909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:57.679909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:57.679909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:57.679909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:57.679909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:57.679909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:57.679909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:57.679909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:57.679909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:57.679909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:57.679909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:57.679909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:07:57.679909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:07:57.679909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:07:57.679909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:07:57.680870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:57.680870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:57.680870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:57.680870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:57.680870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:57.680870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:57.680870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:57.680870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:57.680870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:57.680870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:57.680870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:57.680870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:57.680870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:57.680870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:57.680870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:57.680870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:57.680870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:07:57.681767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:07:57.681767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:07:57.681767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:57.681767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 473/940] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:07:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:57.686871 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8e59a0-2201-43b3-ae77-abf23dcf6c2c None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:57.780119 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:57.790059 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8e59a0-2201-43b3-ae77-abf23dcf6c2c None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:57.828790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6599dd19-0308-4ff8-a06f-f0f2b035cc3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:57.829793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6599dd19-0308-4ff8-a06f-f0f2b035cc3e None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:57.830291 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6599dd19-0308-4ff8-a06f-f0f2b035cc3e None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:57.830766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6599dd19-0308-4ff8-a06f-f0f2b035cc3e None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:57.831617 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6599dd19-0308-4ff8-a06f-f0f2b035cc3e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:57.832110 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6599dd19-0308-4ff8-a06f-f0f2b035cc3e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:57.833698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6599dd19-0308-4ff8-a06f-f0f2b035cc3e None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:57.845867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 468/941] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:07:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:57.855116 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0066dbf6-40ed-4cc5-aab7-457ceb4aabb6 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:57.951502 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:57.956949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0066dbf6-40ed-4cc5-aab7-457ceb4aabb6 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:57.991774 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef91adb-fc24-4577-9b70-81a3119ddc52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:57.992404 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef91adb-fc24-4577-9b70-81a3119ddc52 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:57.992679 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef91adb-fc24-4577-9b70-81a3119ddc52 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:57.992946 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef91adb-fc24-4577-9b70-81a3119ddc52 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:57.999957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef91adb-fc24-4577-9b70-81a3119ddc52 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:58.001382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef91adb-fc24-4577-9b70-81a3119ddc52 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:58.012855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-4ef91adb-fc24-4577-9b70-81a3119ddc52 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:58.012855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:58.012855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:58.012855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:58.012855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:58.012855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:58.012855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:58.012855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:58.012855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:58.012855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:58.012855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:58.012855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:58.012855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:58.012855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:58.012855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:58.013718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:58.013718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:58.013718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:58.013718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:58.013718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:58.013718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:58.013718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:58.013718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:58.013718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:58.013718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:58.013718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:58.013718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:58.013718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:58.013718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:58.013718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:58.013718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:58.013718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:58.014660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:58.014660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:58.014660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:58.014660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:58.014660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:58.014660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:58.014660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:58.014660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:58.014660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 474/942] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:58.019964 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6599dd19-0308-4ff8-a06f-f0f2b035cc3e None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:58.129230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6599dd19-0308-4ff8-a06f-f0f2b035cc3e None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:58.164605 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7157656-6d49-40ad-ae34-526e801e642a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:58.165208 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7157656-6d49-40ad-ae34-526e801e642a None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:58.165482 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7157656-6d49-40ad-ae34-526e801e642a None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:58.165768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7157656-6d49-40ad-ae34-526e801e642a None admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:58.166462 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7157656-6d49-40ad-ae34-526e801e642a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:58.166759 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7157656-6d49-40ad-ae34-526e801e642a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:58.168178 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7157656-6d49-40ad-ae34-526e801e642a None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:58.180864 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 469/943] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:07:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:58.186702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef91adb-fc24-4577-9b70-81a3119ddc52 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:58.191259 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9684fea8-10c3-4eda-962f-d451549b4e23 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:58.191551 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9684fea8-10c3-4eda-962f-d451549b4e23 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:58.191826 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9684fea8-10c3-4eda-962f-d451549b4e23 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:58.193395 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 470/944] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:58 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:58.198721 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d02c9cf-0332-4fe5-8f4c-8de3186d63e8 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:58.199004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d02c9cf-0332-4fe5-8f4c-8de3186d63e8 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:58.199279 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d02c9cf-0332-4fe5-8f4c-8de3186d63e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:58.218376 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-6d02c9cf-0332-4fe5-8f4c-8de3186d63e8 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:58.219577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6d02c9cf-0332-4fe5-8f4c-8de3186d63e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:58.228022 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:58.295916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef91adb-fc24-4577-9b70-81a3119ddc52 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:58.298115 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 471/945] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:58 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 04:07:58.330020 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c25b7d-de3a-4563-aceb-b6b5c6f7c626 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:58.330917 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c25b7d-de3a-4563-aceb-b6b5c6f7c626 None admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:58.331335 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c25b7d-de3a-4563-aceb-b6b5c6f7c626 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:58.331786 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c25b7d-de3a-4563-aceb-b6b5c6f7c626 None admin] PATH_INFO: `/v3/projects/c830a3bfece6456486f4b094076b5364/users/5a2fcc8c9ac44dceadcc30d163bbfa0e/roles/d25f9751bd8d4cb3963a382887a028ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:58.334593 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c25b7d-de3a-4563-aceb-b6b5c6f7c626 None admin] RBAC: Authorizing `identity:create_grant(project_id=c830a3bfece6456486f4b094076b5364, user_id=5a2fcc8c9ac44dceadcc30d163bbfa0e, role_id=d25f9751bd8d4cb3963a382887a028ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:58.336146 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c25b7d-de3a-4563-aceb-b6b5c6f7c626 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:58.351618 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 475/946] 10.222.0.26 () {58 vars in 1439 bytes} [Tue Sep 27 04:07:58 2022] PUT /identity/v3/projects/c830a3bfece6456486f4b094076b5364/users/5a2fcc8c9ac44dceadcc30d163bbfa0e/roles/d25f9751bd8d4cb3963a382887a028ef => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:07:58.546512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55666f9b-47a4-472e-8006-f699faf93582 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:58.547088 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55666f9b-47a4-472e-8006-f699faf93582 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:58.547574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55666f9b-47a4-472e-8006-f699faf93582 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:58.549039 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 472/947] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:58.554828 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ce3eeb-81f3-4e74-ab63-fad1446ccaa8 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:58.555116 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ce3eeb-81f3-4e74-ab63-fad1446ccaa8 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:58.555392 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ce3eeb-81f3-4e74-ab63-fad1446ccaa8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:58.575125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-66ce3eeb-81f3-4e74-ab63-fad1446ccaa8 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:58.576367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-66ce3eeb-81f3-4e74-ab63-fad1446ccaa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:58.588479 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:58.593604 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55666f9b-47a4-472e-8006-f699faf93582 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:58.649256 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 476/948] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:58 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 04:07:58.701803 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55666f9b-47a4-472e-8006-f699faf93582 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:58.735837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e061e1-ff45-4a08-ab12-ed4502200824 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:58.736720 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e061e1-ff45-4a08-ab12-ed4502200824 None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:58.737150 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e061e1-ff45-4a08-ab12-ed4502200824 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:58.737570 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e061e1-ff45-4a08-ab12-ed4502200824 None admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:58.738399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e061e1-ff45-4a08-ab12-ed4502200824 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:58.740026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e061e1-ff45-4a08-ab12-ed4502200824 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:58.743074 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: WARNING keystone.server.flask.common [None req-13e061e1-ff45-4a08-ab12-ed4502200824 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 27 04:07:58.757188 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:58.762175 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.sql.core [None req-13e061e1-ff45-4a08-ab12-ed4502200824 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 27 04:07:58.762175 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 27 04:07:58.762175 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [parameters: {'id': '1549cf26ead54fbea81af9c65e21469d', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 27 04:07:58.762175 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81420) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 27 04:07:58.781937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-13e061e1-ff45-4a08-ab12-ed4502200824 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 1549cf26ead54fbea81af9c65e21469d.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 1549cf26ead54fbea81af9c65e21469d. Sep 27 04:07:58.781937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:58.781937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 27 04:07:58.781937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 27 04:07:58.781937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 27 04:07:58.781937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 27 04:07:58.781937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 27 04:07:58.781937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application result = self._query(query) Sep 27 04:07:58.781937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 27 04:07:58.781937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application conn.query(q) Sep 27 04:07:58.781937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 27 04:07:58.781937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 27 04:07:58.781937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 27 04:07:58.782804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application result.read() Sep 27 04:07:58.782804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 27 04:07:58.782804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 27 04:07:58.782804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 27 04:07:58.782804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 27 04:07:58.782804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 27 04:07:58.782804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 27 04:07:58.782804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 27 04:07:58.782804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 27 04:07:58.782804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 27 04:07:58.782804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:58.782804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 27 04:07:58.782804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:58.782804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:58.782804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 27 04:07:58.782804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 27 04:07:58.782804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Sep 27 04:07:58.782804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return project_ref.to_dict() Sep 27 04:07:58.783747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 27 04:07:58.783747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application next(self.gen) Sep 27 04:07:58.783747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 27 04:07:58.783747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application yield resource Sep 27 04:07:58.783747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 27 04:07:58.783747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application next(self.gen) Sep 27 04:07:58.783747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 27 04:07:58.783747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self.session.rollback() Sep 27 04:07:58.783747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 27 04:07:58.783747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self.force_reraise() Sep 27 04:07:58.783747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 27 04:07:58.783747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise self.value Sep 27 04:07:58.783747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 27 04:07:58.783747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 27 04:07:58.783747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 27 04:07:58.783747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application session.commit() Sep 27 04:07:58.783747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 27 04:07:58.784626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 27 04:07:58.784626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 27 04:07:58.784626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._prepare_impl() Sep 27 04:07:58.784626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 27 04:07:58.784626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self.session.flush() Sep 27 04:07:58.784626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 27 04:07:58.784626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._flush(objects) Sep 27 04:07:58.784626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 27 04:07:58.784626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 27 04:07:58.784626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 27 04:07:58.784626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application compat.raise_( Sep 27 04:07:58.784626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 27 04:07:58.784626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception Sep 27 04:07:58.784626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 27 04:07:58.784626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application flush_context.execute() Sep 27 04:07:58.784626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 27 04:07:58.784626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rec.execute(self) Sep 27 04:07:58.785496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 27 04:07:58.785496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 27 04:07:58.785496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 27 04:07:58.785496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 27 04:07:58.785496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 27 04:07:58.785496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 27 04:07:58.785496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 27 04:07:58.785496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 27 04:07:58.785496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 27 04:07:58.785496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 27 04:07:58.785496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 27 04:07:58.785496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 27 04:07:58.785496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 27 04:07:58.785496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 27 04:07:58.785496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 27 04:07:58.786352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 27 04:07:58.786352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 27 04:07:58.786352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception Sep 27 04:07:58.786352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 27 04:07:58.786352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 27 04:07:58.786352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 27 04:07:58.786352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 27 04:07:58.786352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 27 04:07:58.786352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application result = self._query(query) Sep 27 04:07:58.786352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 27 04:07:58.786352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application conn.query(q) Sep 27 04:07:58.786352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 27 04:07:58.786352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 27 04:07:58.786352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 27 04:07:58.786352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application result.read() Sep 27 04:07:58.786352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 27 04:07:58.786352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 27 04:07:58.787209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 27 04:07:58.787209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 27 04:07:58.787209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 27 04:07:58.787209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 27 04:07:58.787209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 27 04:07:58.787209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 27 04:07:58.787209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 27 04:07:58.787209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 27 04:07:58.787209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [parameters: {'id': '1549cf26ead54fbea81af9c65e21469d', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 27 04:07:58.787209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 27 04:07:58.787209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:58.787209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:07:58.787209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:58.787209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:58.787209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Sep 27 04:07:58.789183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Sep 27 04:07:58.789183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 27 04:07:58.789183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 27 04:07:58.789183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Sep 27 04:07:58.789183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:58.789183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:07:58.789183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:58.789183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:58.789183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:58.789183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:58.789183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:58.789183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:58.789183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:58.789183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:58.789183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:58.789183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:58.789183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:58.790348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:58.790348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Sep 27 04:07:58.790348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Sep 27 04:07:58.790348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:58.790348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:58.790348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Sep 27 04:07:58.790348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.Conflict( Sep 27 04:07:58.790348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 1549cf26ead54fbea81af9c65e21469d. Sep 27 04:07:58.790348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:07:58.790348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 473/949] 10.222.0.26 () {60 vars in 1140 bytes} [Tue Sep 27 04:07:58 2022] POST /identity/v3/projects => generated 262 bytes in 190 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:58.794881 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ce3eeb-81f3-4e74-ab63-fad1446ccaa8 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:58.900236 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ce3eeb-81f3-4e74-ab63-fad1446ccaa8 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:58.936770 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80372a30-4e9e-4120-8462-2900ebd0180d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:58.937519 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80372a30-4e9e-4120-8462-2900ebd0180d None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:58.937824 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80372a30-4e9e-4120-8462-2900ebd0180d None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:58.938119 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80372a30-4e9e-4120-8462-2900ebd0180d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:58.945486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80372a30-4e9e-4120-8462-2900ebd0180d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:58.946949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80372a30-4e9e-4120-8462-2900ebd0180d None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:58.958222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-80372a30-4e9e-4120-8462-2900ebd0180d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:58.958222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:58.958222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:58.958222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:58.958222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:58.958222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:58.958222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:58.958222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:58.958222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:58.958222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:58.958222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:58.958222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:58.958222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:07:58.958222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:07:58.958222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:07:58.959131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:07:58.959131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:58.959131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:58.959131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:58.959131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:58.959131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:58.959131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:58.959131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:58.959131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:58.959131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:58.959131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:58.959131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:58.959131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:58.959131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:58.959131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:58.959131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:58.959131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:58.960070 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:07:58.960070 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:07:58.960070 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:07:58.960070 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:07:58.960070 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:07:58.960070 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:07:58.960070 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:07:58.960070 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:58.960070 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 477/950] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:07:58.965362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e061e1-ff45-4a08-ab12-ed4502200824 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:58.967522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80372a30-4e9e-4120-8462-2900ebd0180d None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:58.978784 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:59.090010 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80372a30-4e9e-4120-8462-2900ebd0180d None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:59.094588 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e061e1-ff45-4a08-ab12-ed4502200824 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:59.129211 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480393a3-c264-4773-a669-758a77ae7f4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:59.130847 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480393a3-c264-4773-a669-758a77ae7f4f None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:59.131382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480393a3-c264-4773-a669-758a77ae7f4f None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:59.131656 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480393a3-c264-4773-a669-758a77ae7f4f None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:59.132279 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480393a3-c264-4773-a669-758a77ae7f4f None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:59.133965 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480393a3-c264-4773-a669-758a77ae7f4f None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:59.139609 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02dd4e6-2a5d-467b-8411-3aa60d34b5ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:59.140489 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02dd4e6-2a5d-467b-8411-3aa60d34b5ac None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:59.140963 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02dd4e6-2a5d-467b-8411-3aa60d34b5ac None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:59.141446 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02dd4e6-2a5d-467b-8411-3aa60d34b5ac None admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:59.142397 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02dd4e6-2a5d-467b-8411-3aa60d34b5ac None admin] RBAC: Adding query filter params (name=service) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:59.142912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02dd4e6-2a5d-467b-8411-3aa60d34b5ac None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:59.143111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-480393a3-c264-4773-a669-758a77ae7f4f None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 04:07:59.143111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:59.143111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:07:59.143111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:07:59.143111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:07:59.143111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:07:59.143111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:07:59.143111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:59.143111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 04:07:59.143111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:59.143111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:07:59.143111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:59.143111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:07:59.143111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:07:59.143111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:07:59.143111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:07:59.143111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:07:59.144051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:07:59.144051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:07:59.144051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:07:59.144051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:07:59.144051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:07:59.144051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:07:59.144051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:07:59.144051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:07:59.144051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:07:59.144051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:07:59.144051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:07:59.144051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:07:59.144051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:07:59.144051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:07:59.144051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:07:59.144051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:07:59.145023 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:07:59.145023 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:07:59.145023 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:07:59.145023 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:07:59.145023 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:07:59.145023 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:07:59.145023 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:07:59.145023 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:07:59.145023 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:07:59.145023 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:07:59.145023 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:07:59.145023 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:07:59.145023 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 04:07:59.145023 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:07:59.145023 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 478/951] 10.222.0.26 () {56 vars in 1126 bytes} [Tue Sep 27 04:07:58 2022] GET /identity/v3/services/object-store => generated 93 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:07:59.147505 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02dd4e6-2a5d-467b-8411-3aa60d34b5ac None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:59.149886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480393a3-c264-4773-a669-758a77ae7f4f None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:59.181146 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 474/952] 10.222.0.26 () {56 vars in 1112 bytes} [Tue Sep 27 04:07:58 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:07:59.234343 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:59.274806 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480393a3-c264-4773-a669-758a77ae7f4f None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:59.314176 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cdab98-487c-43fc-876c-7e144abf95df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:59.315197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cdab98-487c-43fc-876c-7e144abf95df None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:59.315684 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cdab98-487c-43fc-876c-7e144abf95df None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:59.316203 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cdab98-487c-43fc-876c-7e144abf95df None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:59.317088 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cdab98-487c-43fc-876c-7e144abf95df None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:59.317647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cdab98-487c-43fc-876c-7e144abf95df None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:59.319326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cdab98-487c-43fc-876c-7e144abf95df None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:59.325966 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 479/953] 10.222.0.26 () {56 vars in 1122 bytes} [Tue Sep 27 04:07:59 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:07:59.333772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02dd4e6-2a5d-467b-8411-3aa60d34b5ac None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:59.400135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:59.448338 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02dd4e6-2a5d-467b-8411-3aa60d34b5ac None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:59.455591 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb26fa0-59dd-4a86-8dc9-d38b10066d3f None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:59.455884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb26fa0-59dd-4a86-8dc9-d38b10066d3f None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:59.456164 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb26fa0-59dd-4a86-8dc9-d38b10066d3f None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:59.457620 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 480/954] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:07:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:07:59.462645 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955fac82-1d7c-4dd1-86bb-70d646e3596a None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:59.462999 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955fac82-1d7c-4dd1-86bb-70d646e3596a None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:59.463280 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955fac82-1d7c-4dd1-86bb-70d646e3596a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:59.484998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38793b0d-825f-4b73-958d-9c55f9a3f29e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:59.485694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38793b0d-825f-4b73-958d-9c55f9a3f29e None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:59.485921 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38793b0d-825f-4b73-958d-9c55f9a3f29e None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:59.486153 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38793b0d-825f-4b73-958d-9c55f9a3f29e None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:59.486778 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38793b0d-825f-4b73-958d-9c55f9a3f29e None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:07:59.487074 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38793b0d-825f-4b73-958d-9c55f9a3f29e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:59.488901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38793b0d-825f-4b73-958d-9c55f9a3f29e None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:59.489090 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-955fac82-1d7c-4dd1-86bb-70d646e3596a None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:07:59.490982 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-955fac82-1d7c-4dd1-86bb-70d646e3596a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:07:59.495417 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 475/955] 10.222.0.26 () {56 vars in 1122 bytes} [Tue Sep 27 04:07:59 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:07:59.500943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38793b0d-825f-4b73-958d-9c55f9a3f29e None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:59.577970 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 481/956] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:07:59 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:07:59.614333 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38793b0d-825f-4b73-958d-9c55f9a3f29e None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:07:59.638032 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:59.651322 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a8324a-efde-47d2-aa5a-5d8dcfa20d60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:07:59.651987 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a8324a-efde-47d2-aa5a-5d8dcfa20d60 None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:07:59.652265 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a8324a-efde-47d2-aa5a-5d8dcfa20d60 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:07:59.652534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a8324a-efde-47d2-aa5a-5d8dcfa20d60 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:07:59.653258 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a8324a-efde-47d2-aa5a-5d8dcfa20d60 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:07:59.654664 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a8324a-efde-47d2-aa5a-5d8dcfa20d60 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:07:59.677091 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 476/957] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:07:59 2022] POST /identity/v3/endpoints => generated 349 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:07:59.864556 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955fac82-1d7c-4dd1-86bb-70d646e3596a None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:07:59.931738 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:07:59.970984 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955fac82-1d7c-4dd1-86bb-70d646e3596a None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:00.011289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272ea55b-2f33-470a-92ee-4b3d9b0aa730 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:00.011918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272ea55b-2f33-470a-92ee-4b3d9b0aa730 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:00.012190 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272ea55b-2f33-470a-92ee-4b3d9b0aa730 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:00.012457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272ea55b-2f33-470a-92ee-4b3d9b0aa730 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:00.013098 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-272ea55b-2f33-470a-92ee-4b3d9b0aa730 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:00.014509 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-272ea55b-2f33-470a-92ee-4b3d9b0aa730 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:00.024295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-272ea55b-2f33-470a-92ee-4b3d9b0aa730 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 27 04:08:00.024295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:00.024295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:08:00.024295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:08:00.024295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:08:00.024295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:08:00.024295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:08:00.024295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:08:00.024295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 27 04:08:00.024295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:00.024295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:08:00.024295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:00.024295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:00.024295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:00.024295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:00.024295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:00.024295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:00.025382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:00.025382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:00.025382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:00.025382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:00.025382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:00.025382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:00.025382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:08:00.025382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:08:00.025382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:08:00.025382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:08:00.025382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:00.025382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:00.025382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:00.025382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:00.025382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:00.025382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:00.026308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:00.026308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:00.026308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:00.026308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:00.026308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:00.026308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:00.026308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:00.026308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:00.026308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:00.026308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:00.026308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:08:00.026308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:08:00.026308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 27 04:08:00.026308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:00.026308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 482/958] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:07:59 2022] GET /identity/v3/services/compute => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:08:00.031298 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a8324a-efde-47d2-aa5a-5d8dcfa20d60 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:00.135361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a8324a-efde-47d2-aa5a-5d8dcfa20d60 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:00.170052 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc391262-f49a-489d-8fa1-243cf2621eaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:00.170658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc391262-f49a-489d-8fa1-243cf2621eaf None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:00.170935 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc391262-f49a-489d-8fa1-243cf2621eaf None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:00.171205 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc391262-f49a-489d-8fa1-243cf2621eaf None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:00.171831 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc391262-f49a-489d-8fa1-243cf2621eaf None admin] RBAC: Adding query filter params (name=compute) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:00.172129 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc391262-f49a-489d-8fa1-243cf2621eaf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:00.173483 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc391262-f49a-489d-8fa1-243cf2621eaf None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:00.178674 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 477/959] 10.222.0.26 () {56 vars in 1112 bytes} [Tue Sep 27 04:08:00 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:00.184269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272ea55b-2f33-470a-92ee-4b3d9b0aa730 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:00.290443 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272ea55b-2f33-470a-92ee-4b3d9b0aa730 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:00.318669 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:00.330134 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60447df6-91e7-4a08-b633-65bcc9749bfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:00.330795 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60447df6-91e7-4a08-b633-65bcc9749bfb None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:00.331078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60447df6-91e7-4a08-b633-65bcc9749bfb None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:00.331352 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60447df6-91e7-4a08-b633-65bcc9749bfb None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:00.331985 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60447df6-91e7-4a08-b633-65bcc9749bfb None admin] RBAC: Adding query filter params (type=compute) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:00.332285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60447df6-91e7-4a08-b633-65bcc9749bfb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:00.333670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60447df6-91e7-4a08-b633-65bcc9749bfb None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:00.338497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 483/960] 10.222.0.26 () {56 vars in 1112 bytes} [Tue Sep 27 04:08:00 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:00.649513 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7aae3c-4894-430f-af02-a27bb3a32dd2 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:00.649883 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7aae3c-4894-430f-af02-a27bb3a32dd2 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:00.650121 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7aae3c-4894-430f-af02-a27bb3a32dd2 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:00.651716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 478/961] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:08:00.656985 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d516cf-dca5-4747-b223-51907279123c None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:00.657216 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d516cf-dca5-4747-b223-51907279123c None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:00.657479 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d516cf-dca5-4747-b223-51907279123c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:00.676052 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-63d516cf-dca5-4747-b223-51907279123c None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:00.677859 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-63d516cf-dca5-4747-b223-51907279123c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:00.749855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 484/962] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:00 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:08:01.019442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7aae3c-4894-430f-af02-a27bb3a32dd2 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:01.066577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:01.131435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7aae3c-4894-430f-af02-a27bb3a32dd2 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:01.166873 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c519c8-419b-4e00-8bf6-9842bd193120 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:01.168166 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c519c8-419b-4e00-8bf6-9842bd193120 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:01.168617 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c519c8-419b-4e00-8bf6-9842bd193120 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:01.169044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c519c8-419b-4e00-8bf6-9842bd193120 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:01.181999 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c519c8-419b-4e00-8bf6-9842bd193120 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:08:01.187082 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-e7c519c8-419b-4e00-8bf6-9842bd193120 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 27 04:08:01.187082 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:01.187082 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:01.187082 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:01.187082 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:01.187082 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:01.187082 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:01.187082 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:01.187082 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:01.187082 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:01.187082 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:01.187082 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:01.187082 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:08:01.187082 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:08:01.187082 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:08:01.187082 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:08:01.188192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:08:01.188192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:08:01.188192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:08:01.188192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:08:01.188192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:01.188192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:01.188192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:01.188192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:01.188192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:01.188192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:01.188192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:01.188192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:01.188192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:01.188192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:01.188192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:01.188192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:01.189231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:01.189231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:01.189231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:01.189231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:01.189231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:08:01.189231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:08:01.189231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:08:01.189231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:08:01.189231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:08:01.189231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:08:01.189231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 27 04:08:01.189231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:01.191403 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 479/963] 10.222.0.26 () {56 vars in 1120 bytes} [Tue Sep 27 04:08:01 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:08:01.201785 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d516cf-dca5-4747-b223-51907279123c None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:01.302109 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d516cf-dca5-4747-b223-51907279123c None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:01.320342 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:01.339419 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000009b5-1da5-4d51-8a74-b7c15ce4ef12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:01.340017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000009b5-1da5-4d51-8a74-b7c15ce4ef12 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:01.340290 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000009b5-1da5-4d51-8a74-b7c15ce4ef12 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:01.340561 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000009b5-1da5-4d51-8a74-b7c15ce4ef12 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:01.341219 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000009b5-1da5-4d51-8a74-b7c15ce4ef12 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:01.341513 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000009b5-1da5-4d51-8a74-b7c15ce4ef12 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:01.344269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000009b5-1da5-4d51-8a74-b7c15ce4ef12 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:01.357870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 485/964] 10.222.0.26 () {56 vars in 1115 bytes} [Tue Sep 27 04:08:01 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:01.359453 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c519c8-419b-4e00-8bf6-9842bd193120 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:01.475247 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c519c8-419b-4e00-8bf6-9842bd193120 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:01.477123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:01.512023 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2e810c-be94-4303-ba37-79f194ab8165 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:01.513059 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2e810c-be94-4303-ba37-79f194ab8165 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:01.513560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2e810c-be94-4303-ba37-79f194ab8165 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:01.514075 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2e810c-be94-4303-ba37-79f194ab8165 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:01.527529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2e810c-be94-4303-ba37-79f194ab8165 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:01.529366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2e810c-be94-4303-ba37-79f194ab8165 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:01.545697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-af2e810c-be94-4303-ba37-79f194ab8165 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:08:01.545697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:01.545697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:01.545697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:01.545697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:01.545697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:01.545697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:01.545697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:01.545697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:01.545697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:01.545697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:01.545697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:01.545697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:08:01.545697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:08:01.545697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:08:01.545697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:08:01.546842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:01.546842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:01.546842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:01.546842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:01.546842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:01.546842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:01.546842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:01.546842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:01.546842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:01.546842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:01.546842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:01.546842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:01.546842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:01.546842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:01.546842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:01.546842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:01.546842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:08:01.547763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:08:01.547763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:08:01.547763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:01.549857 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 480/965] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:08:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:08:01.557848 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000009b5-1da5-4d51-8a74-b7c15ce4ef12 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:01.644558 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd38ab5e-d816-49b3-984b-3e353ded278c None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:01.644880 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd38ab5e-d816-49b3-984b-3e353ded278c None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:01.645100 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd38ab5e-d816-49b3-984b-3e353ded278c None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:01.647275 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 481/966] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:08:01.652147 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d625eec9-f4a3-4f1c-b7e8-d0c937044fba None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:01.652435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d625eec9-f4a3-4f1c-b7e8-d0c937044fba None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:01.652708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d625eec9-f4a3-4f1c-b7e8-d0c937044fba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:01.664372 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000009b5-1da5-4d51-8a74-b7c15ce4ef12 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:01.672883 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-d625eec9-f4a3-4f1c-b7e8-d0c937044fba None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:01.674623 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d625eec9-f4a3-4f1c-b7e8-d0c937044fba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:01.698319 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de4e5f7-b82f-4b7a-b524-fa717e89bf84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:01.698923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de4e5f7-b82f-4b7a-b524-fa717e89bf84 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:01.699196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de4e5f7-b82f-4b7a-b524-fa717e89bf84 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:01.699460 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de4e5f7-b82f-4b7a-b524-fa717e89bf84 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:01.700093 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de4e5f7-b82f-4b7a-b524-fa717e89bf84 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:01.700388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de4e5f7-b82f-4b7a-b524-fa717e89bf84 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:01.701727 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de4e5f7-b82f-4b7a-b524-fa717e89bf84 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:01.714930 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 486/967] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:08:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:01.727424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de4e5f7-b82f-4b7a-b524-fa717e89bf84 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:01.753827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 482/968] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:01 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 04:08:01.771617 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:01.836535 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de4e5f7-b82f-4b7a-b524-fa717e89bf84 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:01.870614 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee5186c-0e92-42af-bb9f-36147e300c69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:01.871236 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee5186c-0e92-42af-bb9f-36147e300c69 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:01.871514 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee5186c-0e92-42af-bb9f-36147e300c69 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:01.871774 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee5186c-0e92-42af-bb9f-36147e300c69 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:01.880610 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee5186c-0e92-42af-bb9f-36147e300c69 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:01.882106 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee5186c-0e92-42af-bb9f-36147e300c69 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:01.895759 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-4ee5186c-0e92-42af-bb9f-36147e300c69 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 27 04:08:01.895759 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:01.895759 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:01.895759 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:01.895759 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:01.895759 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:01.895759 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:01.895759 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:01.895759 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:01.895759 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:01.895759 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:01.895759 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:01.895759 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:08:01.895759 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:08:01.895759 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:08:01.895759 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:08:01.896796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:01.896796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:01.896796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:08:01.896796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:08:01.896796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:08:01.896796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:08:01.896796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:01.896796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:01.896796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:01.896796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:01.896796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:01.896796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:01.896796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:01.896796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:01.896796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:01.896796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:01.896796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:01.896796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:01.897737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:01.897737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:01.897737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:08:01.897737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:08:01.897737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:08:01.897737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:08:01.897737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:08:01.897737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:08:01.897737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 27 04:08:01.897737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:01.897737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 487/969] 10.222.0.26 () {56 vars in 1117 bytes} [Tue Sep 27 04:08:01 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:08:01.903152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d625eec9-f4a3-4f1c-b7e8-d0c937044fba None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:01.990768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:02.003972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a00189e-1a2b-4eff-baad-742e5af512f5 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:02.004274 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a00189e-1a2b-4eff-baad-742e5af512f5 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:02.004520 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a00189e-1a2b-4eff-baad-742e5af512f5 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:02.006993 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 488/970] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:08:02.009903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d625eec9-f4a3-4f1c-b7e8-d0c937044fba None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:02.011838 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c23c54f-3fed-42d4-b8e9-5efc9a0a2c85 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:02.012118 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c23c54f-3fed-42d4-b8e9-5efc9a0a2c85 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:02.012391 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c23c54f-3fed-42d4-b8e9-5efc9a0a2c85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:02.031259 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-9c23c54f-3fed-42d4-b8e9-5efc9a0a2c85 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:02.032427 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-9c23c54f-3fed-42d4-b8e9-5efc9a0a2c85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:02.050768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9468c6-89aa-4606-a170-a11e381356ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:02.051379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9468c6-89aa-4606-a170-a11e381356ab None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:02.051658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9468c6-89aa-4606-a170-a11e381356ab None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:02.052459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9468c6-89aa-4606-a170-a11e381356ab None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:02.053181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9468c6-89aa-4606-a170-a11e381356ab None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:02.053480 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9468c6-89aa-4606-a170-a11e381356ab None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:02.056220 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9468c6-89aa-4606-a170-a11e381356ab None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:02.074806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 483/971] 10.222.0.26 () {56 vars in 1149 bytes} [Tue Sep 27 04:08:01 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:02.075681 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9468c6-89aa-4606-a170-a11e381356ab None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:02.107667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 489/972] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:02 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 04:08:02.109157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c23c54f-3fed-42d4-b8e9-5efc9a0a2c85 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:02.150826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:02.187867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9468c6-89aa-4606-a170-a11e381356ab None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:02.225758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3954a2-a27d-4948-8c29-41957056d9a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:02.227191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3954a2-a27d-4948-8c29-41957056d9a1 None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:02.227812 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3954a2-a27d-4948-8c29-41957056d9a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:02.228544 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3954a2-a27d-4948-8c29-41957056d9a1 None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:02.230036 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3954a2-a27d-4948-8c29-41957056d9a1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:02.231606 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3954a2-a27d-4948-8c29-41957056d9a1 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:02.246290 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c23c54f-3fed-42d4-b8e9-5efc9a0a2c85 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:02.247806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.sql.core [None req-1b3954a2-a27d-4948-8c29-41957056d9a1 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 27 04:08:02.247806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 27 04:08:02.247806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [parameters: {'user_id': 'f9fa59f469024404a52e5be3036c746c', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 27 04:08:02.247806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81420) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 27 04:08:02.256310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-1b3954a2-a27d-4948-8c29-41957056d9a1 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 27 04:08:02.256310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:02.256310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 27 04:08:02.256310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 27 04:08:02.256310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 27 04:08:02.256310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 27 04:08:02.256310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 27 04:08:02.256310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application result = self._query(query) Sep 27 04:08:02.256310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 27 04:08:02.256310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application conn.query(q) Sep 27 04:08:02.256310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 27 04:08:02.256310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 27 04:08:02.256310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 27 04:08:02.256310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application result.read() Sep 27 04:08:02.256310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 27 04:08:02.257230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 27 04:08:02.257230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 27 04:08:02.257230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 27 04:08:02.257230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 27 04:08:02.257230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 27 04:08:02.257230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 27 04:08:02.257230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 27 04:08:02.257230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 27 04:08:02.257230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:02.257230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 27 04:08:02.257230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:02.257230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:02.257230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 27 04:08:02.257230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 27 04:08:02.257230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Sep 27 04:08:02.257230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Sep 27 04:08:02.257230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 27 04:08:02.257230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application next(self.gen) Sep 27 04:08:02.258277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 27 04:08:02.258277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application yield resource Sep 27 04:08:02.258277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 27 04:08:02.258277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application next(self.gen) Sep 27 04:08:02.258277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 27 04:08:02.258277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self.session.rollback() Sep 27 04:08:02.258277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 27 04:08:02.258277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self.force_reraise() Sep 27 04:08:02.258277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 27 04:08:02.258277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise self.value Sep 27 04:08:02.258277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 27 04:08:02.258277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 27 04:08:02.258277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 27 04:08:02.258277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application session.commit() Sep 27 04:08:02.258277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 27 04:08:02.258277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 27 04:08:02.258277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 27 04:08:02.259193 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._prepare_impl() Sep 27 04:08:02.259193 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 27 04:08:02.259193 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self.session.flush() Sep 27 04:08:02.259193 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 27 04:08:02.259193 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._flush(objects) Sep 27 04:08:02.259193 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 27 04:08:02.259193 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 27 04:08:02.259193 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 27 04:08:02.259193 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application compat.raise_( Sep 27 04:08:02.259193 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 27 04:08:02.259193 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception Sep 27 04:08:02.259193 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 27 04:08:02.259193 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application flush_context.execute() Sep 27 04:08:02.259193 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 27 04:08:02.259193 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rec.execute(self) Sep 27 04:08:02.259193 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 27 04:08:02.259193 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 27 04:08:02.260097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 27 04:08:02.260097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 27 04:08:02.260097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Sep 27 04:08:02.260097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application result = connection._execute_20( Sep 27 04:08:02.260097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 27 04:08:02.260097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 27 04:08:02.260097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 27 04:08:02.260097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 27 04:08:02.260097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 27 04:08:02.260097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 27 04:08:02.260097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 27 04:08:02.260097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 27 04:08:02.260097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 27 04:08:02.260097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 27 04:08:02.260097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 27 04:08:02.260097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception Sep 27 04:08:02.260944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 27 04:08:02.260944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 27 04:08:02.260944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 27 04:08:02.260944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 27 04:08:02.260944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 27 04:08:02.260944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application result = self._query(query) Sep 27 04:08:02.260944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 27 04:08:02.260944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application conn.query(q) Sep 27 04:08:02.260944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 27 04:08:02.260944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 27 04:08:02.260944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 27 04:08:02.260944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application result.read() Sep 27 04:08:02.260944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 27 04:08:02.260944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 27 04:08:02.260944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 27 04:08:02.260944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 27 04:08:02.262026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 27 04:08:02.262026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 27 04:08:02.262026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 27 04:08:02.262026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 27 04:08:02.262026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 27 04:08:02.262026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 27 04:08:02.262026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [parameters: {'user_id': 'f9fa59f469024404a52e5be3036c746c', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 27 04:08:02.262026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 27 04:08:02.262026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:02.262026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:08:02.262026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:02.262026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:02.262026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:02.262026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:02.262026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:02.263122 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:02.263122 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:02.263122 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:02.263122 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:02.263122 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:02.263122 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:02.263122 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:02.263122 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Sep 27 04:08:02.263122 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Sep 27 04:08:02.263122 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:02.263122 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:02.263122 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:08:02.263122 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:08:02.263122 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:08:02.263122 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:08:02.263122 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Sep 27 04:08:02.263122 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Sep 27 04:08:02.264958 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Sep 27 04:08:02.264958 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Sep 27 04:08:02.264958 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 27 04:08:02.264958 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 27 04:08:02.264958 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 27 04:08:02.264958 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:02.267789 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 484/973] 10.222.0.26 () {60 vars in 1131 bytes} [Tue Sep 27 04:08:02 2022] POST /identity/v3/users => generated 160 bytes in 192 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:08:02.274226 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3954a2-a27d-4948-8c29-41957056d9a1 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:02.279652 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:02.291594 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efd9914-06e2-4442-a48f-74d52a8bc3ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:02.292580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efd9914-06e2-4442-a48f-74d52a8bc3ec None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:02.293096 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efd9914-06e2-4442-a48f-74d52a8bc3ec None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:02.293587 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efd9914-06e2-4442-a48f-74d52a8bc3ec None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:02.370254 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1efd9914-06e2-4442-a48f-74d52a8bc3ec None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:02.371621 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1efd9914-06e2-4442-a48f-74d52a8bc3ec None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:02.400209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3954a2-a27d-4948-8c29-41957056d9a1 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:02.408053 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:02.438254 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565c5583-2f42-4872-b48e-455dc45bc7db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:02.439177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565c5583-2f42-4872-b48e-455dc45bc7db None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:02.439455 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565c5583-2f42-4872-b48e-455dc45bc7db None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:02.439732 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565c5583-2f42-4872-b48e-455dc45bc7db None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:02.448404 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565c5583-2f42-4872-b48e-455dc45bc7db None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:02.450087 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565c5583-2f42-4872-b48e-455dc45bc7db None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:02.462928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-565c5583-2f42-4872-b48e-455dc45bc7db None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 27 04:08:02.462928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:02.462928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:02.462928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:02.462928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:02.462928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:02.462928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:02.462928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:02.462928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:02.462928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:02.462928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:02.462928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:02.462928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:08:02.462928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:08:02.462928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:08:02.462928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:08:02.464097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:02.464097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:02.464097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:08:02.464097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:08:02.464097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:08:02.464097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:08:02.464097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:02.464097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:02.464097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:02.464097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:02.464097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:02.464097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:02.464097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:02.464097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:02.464097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:02.464097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:02.464097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:02.464097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:02.465387 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:02.465387 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:02.465387 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:08:02.465387 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:08:02.465387 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:08:02.465387 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:08:02.465387 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:08:02.465387 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:08:02.465387 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 27 04:08:02.465387 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:02.468845 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 485/974] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:08:02 2022] GET /identity/v3/users/cinder => generated 84 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:08:02.469669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565c5583-2f42-4872-b48e-455dc45bc7db None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:02.494821 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 490/975] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:08:02 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 04:08:02.496332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efd9914-06e2-4442-a48f-74d52a8bc3ec None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:02.535504 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:02.574347 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565c5583-2f42-4872-b48e-455dc45bc7db None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:02.599851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efd9914-06e2-4442-a48f-74d52a8bc3ec None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:02.606607 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9ac45f-e0a0-41af-b3a9-7b4c29479f41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:02.607226 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9ac45f-e0a0-41af-b3a9-7b4c29479f41 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:02.607503 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9ac45f-e0a0-41af-b3a9-7b4c29479f41 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:02.607773 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9ac45f-e0a0-41af-b3a9-7b4c29479f41 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:02.608343 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c9ac45f-e0a0-41af-b3a9-7b4c29479f41 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:02.609691 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c9ac45f-e0a0-41af-b3a9-7b4c29479f41 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:02.618792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-0c9ac45f-e0a0-41af-b3a9-7b4c29479f41 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 04:08:02.618792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:02.618792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:08:02.618792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:08:02.618792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:08:02.618792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:08:02.618792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:08:02.618792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:08:02.618792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 04:08:02.618792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:02.618792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:08:02.618792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:02.618792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:02.618792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:02.618792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:02.618792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:02.618792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:02.619923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:02.619923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:02.619923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:02.619923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:02.619923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:02.619923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:02.619923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:08:02.619923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:08:02.619923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:08:02.619923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:08:02.619923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:02.619923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:02.619923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:02.619923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:02.619923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:02.619923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:02.621214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:02.621214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:02.621214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:02.621214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:02.621214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:02.621214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:02.621214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:02.621214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:02.621214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:02.621214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:02.621214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:08:02.621214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:08:02.621214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 04:08:02.621214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:02.621214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 486/976] 10.222.0.26 () {56 vars in 1126 bytes} [Tue Sep 27 04:08:02 2022] GET /identity/v3/services/object-store => generated 93 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:08:02.622067 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9ac45f-e0a0-41af-b3a9-7b4c29479f41 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:02.631533 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:02.637721 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0103ee97-7f11-4c24-9880-98e1dbcd372b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:02.638357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0103ee97-7f11-4c24-9880-98e1dbcd372b None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:02.638635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0103ee97-7f11-4c24-9880-98e1dbcd372b None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:02.638906 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0103ee97-7f11-4c24-9880-98e1dbcd372b None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:02.639598 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0103ee97-7f11-4c24-9880-98e1dbcd372b None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:02.639901 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0103ee97-7f11-4c24-9880-98e1dbcd372b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:02.641322 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0103ee97-7f11-4c24-9880-98e1dbcd372b None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:02.674890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 491/977] 10.222.0.26 () {56 vars in 1101 bytes} [Tue Sep 27 04:08:02 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:02.676096 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a866937-ab00-4c6b-8929-e2aadab99cae None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:02.676381 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a866937-ab00-4c6b-8929-e2aadab99cae None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:02.676652 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a866937-ab00-4c6b-8929-e2aadab99cae None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:02.678659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 492/978] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:08:02.683562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a866937-ab00-4c6b-8929-e2aadab99cae None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:02.728596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9ac45f-e0a0-41af-b3a9-7b4c29479f41 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:02.759590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:02.768188 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299da455-d986-4156-9b44-8dd95f05d244 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:02.768843 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299da455-d986-4156-9b44-8dd95f05d244 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:02.769124 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299da455-d986-4156-9b44-8dd95f05d244 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:02.769401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299da455-d986-4156-9b44-8dd95f05d244 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:02.780831 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299da455-d986-4156-9b44-8dd95f05d244 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:02.782368 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299da455-d986-4156-9b44-8dd95f05d244 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:02.798413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-299da455-d986-4156-9b44-8dd95f05d244 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:08:02.798413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:02.798413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:02.798413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:02.798413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:02.798413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:02.798413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:02.798413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:02.798413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:02.798413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:02.798413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:02.798413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:02.798413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:08:02.798413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:08:02.798413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:08:02.798413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:08:02.799476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:02.799476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:02.799476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:02.799476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:02.799476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:02.799476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:02.799476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:02.799476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:02.799476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:02.799476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:02.799476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:02.799476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:02.799476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:02.799476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:02.799476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:02.799476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:02.799476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:08:02.800358 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:08:02.800358 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:08:02.800358 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:02.800358 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 487/979] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:08:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:08:02.800935 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d37262d-92e4-4754-8868-be45f57d6033 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:02.801219 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d37262d-92e4-4754-8868-be45f57d6033 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:02.801500 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d37262d-92e4-4754-8868-be45f57d6033 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:02.814169 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a866937-ab00-4c6b-8929-e2aadab99cae None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:02.821626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-1d37262d-92e4-4754-8868-be45f57d6033 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:02.823220 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-1d37262d-92e4-4754-8868-be45f57d6033 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:02.857246 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b86b518-0644-4654-8672-71f3d86b233a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:02.857246 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b86b518-0644-4654-8672-71f3d86b233a None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:02.857559 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b86b518-0644-4654-8672-71f3d86b233a None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:02.857636 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b86b518-0644-4654-8672-71f3d86b233a None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:02.858582 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b86b518-0644-4654-8672-71f3d86b233a None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:02.858582 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b86b518-0644-4654-8672-71f3d86b233a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:02.860452 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b86b518-0644-4654-8672-71f3d86b233a None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:02.866248 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 493/980] 10.222.0.26 () {56 vars in 1122 bytes} [Tue Sep 27 04:08:02 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:02.867385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b86b518-0644-4654-8672-71f3d86b233a None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:02.896703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 488/981] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:02 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:08:02.897682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d37262d-92e4-4754-8868-be45f57d6033 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:02.940689 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:03.005691 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b86b518-0644-4654-8672-71f3d86b233a None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:03.024230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d37262d-92e4-4754-8868-be45f57d6033 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:03.035036 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:03.044770 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a132cdf-01a6-422f-ac8f-2727fd5adf67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:03.045661 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a132cdf-01a6-422f-ac8f-2727fd5adf67 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:03.046224 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a132cdf-01a6-422f-ac8f-2727fd5adf67 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:03.046591 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a132cdf-01a6-422f-ac8f-2727fd5adf67 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:03.047241 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a132cdf-01a6-422f-ac8f-2727fd5adf67 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:03.047541 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a132cdf-01a6-422f-ac8f-2727fd5adf67 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:03.048883 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a132cdf-01a6-422f-ac8f-2727fd5adf67 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:03.061606 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 494/982] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:08:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:03.061963 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f49466-a258-4725-934e-fa7c7144d772 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:03.062730 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f49466-a258-4725-934e-fa7c7144d772 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:03.063149 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f49466-a258-4725-934e-fa7c7144d772 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:03.063563 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f49466-a258-4725-934e-fa7c7144d772 None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:03.064327 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f49466-a258-4725-934e-fa7c7144d772 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:03.064783 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f49466-a258-4725-934e-fa7c7144d772 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:03.066386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f49466-a258-4725-934e-fa7c7144d772 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:03.067071 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a132cdf-01a6-422f-ac8f-2727fd5adf67 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:03.075004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 489/983] 10.222.0.26 () {56 vars in 1122 bytes} [Tue Sep 27 04:08:02 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:03.080751 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f49466-a258-4725-934e-fa7c7144d772 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:03.176447 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a132cdf-01a6-422f-ac8f-2727fd5adf67 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:03.183815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f49466-a258-4725-934e-fa7c7144d772 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:03.190014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:03.211923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa4c2c5-312f-40e5-95e1-0e061d5c03c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:03.212545 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa4c2c5-312f-40e5-95e1-0e061d5c03c1 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:03.212822 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa4c2c5-312f-40e5-95e1-0e061d5c03c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:03.213097 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa4c2c5-312f-40e5-95e1-0e061d5c03c1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:03.216830 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df02be0-61d5-40d5-9174-8d8fcb9d39cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:03.217426 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df02be0-61d5-40d5-9174-8d8fcb9d39cb None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:03.217705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df02be0-61d5-40d5-9174-8d8fcb9d39cb None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:03.218009 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df02be0-61d5-40d5-9174-8d8fcb9d39cb None admin] PATH_INFO: `/v3/endpoints` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:03.218711 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df02be0-61d5-40d5-9174-8d8fcb9d39cb None admin] RBAC: Adding query filter params (interface=admin, service_id=55a0fac13c71463bad022337ce1dcd45, region_id=RegionOne) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:03.219018 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df02be0-61d5-40d5-9174-8d8fcb9d39cb None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:03.219581 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa4c2c5-312f-40e5-95e1-0e061d5c03c1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:03.220367 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df02be0-61d5-40d5-9174-8d8fcb9d39cb None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:03.220988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa4c2c5-312f-40e5-95e1-0e061d5c03c1 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:03.225815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 490/984] 10.222.0.26 () {56 vars in 1249 bytes} [Tue Sep 27 04:08:03 2022] GET /identity/v3/endpoints?service_id=55a0fac13c71463bad022337ce1dcd45&interface=admin®ion_id=RegionOne => generated 193 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:08:03.226577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df02be0-61d5-40d5-9174-8d8fcb9d39cb None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:03.238542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-6fa4c2c5-312f-40e5-95e1-0e061d5c03c1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:08:03.238542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:03.238542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:03.238542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:03.238542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:03.238542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:03.238542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:03.238542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:03.238542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:03.238542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:03.238542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:03.238542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:03.238542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:08:03.238542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:08:03.238542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:08:03.240173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:08:03.240173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:03.240173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:03.240173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:03.240173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:03.240173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:03.240173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:03.240173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:03.240173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:03.240173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:03.240173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:03.240173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:03.240173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:03.240173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:03.240173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:03.240173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:03.240173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:03.241067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:08:03.241067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:08:03.241067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:08:03.241067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:08:03.241067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:08:03.241067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:08:03.241067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:08:03.241067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:03.241067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 495/985] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:08:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:08:03.241587 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa4c2c5-312f-40e5-95e1-0e061d5c03c1 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:03.269263 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:03.330729 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df02be0-61d5-40d5-9174-8d8fcb9d39cb None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:03.343630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa4c2c5-312f-40e5-95e1-0e061d5c03c1 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:03.364804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6a674d-536c-420f-83a5-36ad19a979eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:03.365403 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6a674d-536c-420f-83a5-36ad19a979eb None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:03.365683 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6a674d-536c-420f-83a5-36ad19a979eb None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:03.365983 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6a674d-536c-420f-83a5-36ad19a979eb None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:03.366626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6a674d-536c-420f-83a5-36ad19a979eb None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:03.367976 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6a674d-536c-420f-83a5-36ad19a979eb None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:03.376913 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8bf712-fda3-4ce0-93f2-a5ac898eb57e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:03.377582 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8bf712-fda3-4ce0-93f2-a5ac898eb57e None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:03.378033 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8bf712-fda3-4ce0-93f2-a5ac898eb57e None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:03.378298 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8bf712-fda3-4ce0-93f2-a5ac898eb57e None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:03.378902 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8bf712-fda3-4ce0-93f2-a5ac898eb57e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:03.380510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8bf712-fda3-4ce0-93f2-a5ac898eb57e None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:03.382241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 491/986] 10.222.0.26 () {60 vars in 1141 bytes} [Tue Sep 27 04:08:03 2022] POST /identity/v3/services => generated 242 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:08:03.383055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6a674d-536c-420f-83a5-36ad19a979eb None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:03.387212 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 496/987] 10.222.0.26 () {56 vars in 1087 bytes} [Tue Sep 27 04:08:03 2022] GET /identity/v3/services => generated 1727 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:08:03.510643 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:03.525101 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6a674d-536c-420f-83a5-36ad19a979eb None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:03.564322 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eacf2075-3071-4706-8dac-fb006689e7e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:03.564939 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacf2075-3071-4706-8dac-fb006689e7e1 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:03.565209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacf2075-3071-4706-8dac-fb006689e7e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:03.565476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacf2075-3071-4706-8dac-fb006689e7e1 None admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:03.566203 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eacf2075-3071-4706-8dac-fb006689e7e1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:03.566505 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eacf2075-3071-4706-8dac-fb006689e7e1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:03.568653 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eacf2075-3071-4706-8dac-fb006689e7e1 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:03.582734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 492/988] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:08:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:03.589913 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8bf712-fda3-4ce0-93f2-a5ac898eb57e None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:03.632703 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:03.698144 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8bf712-fda3-4ce0-93f2-a5ac898eb57e None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:03.731568 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254a9724-017b-4c69-b366-65adba0a9a9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:03.732156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254a9724-017b-4c69-b366-65adba0a9a9c None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:03.732435 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254a9724-017b-4c69-b366-65adba0a9a9c None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:03.732705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254a9724-017b-4c69-b366-65adba0a9a9c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:03.733430 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254a9724-017b-4c69-b366-65adba0a9a9c None admin] RBAC: Adding query filter params (role.id=d25f9751bd8d4cb3963a382887a028ef, scope.project.id=c830a3bfece6456486f4b094076b5364, user.id=5a2fcc8c9ac44dceadcc30d163bbfa0e) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:03.733733 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254a9724-017b-4c69-b366-65adba0a9a9c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:03.735428 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254a9724-017b-4c69-b366-65adba0a9a9c None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:03.741698 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 497/989] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:08:03 2022] GET /identity/v3/role_assignments?user.id=5a2fcc8c9ac44dceadcc30d163bbfa0e&scope.project.id=c830a3bfece6456486f4b094076b5364&role.id=d25f9751bd8d4cb3963a382887a028ef => generated 608 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:05.568616 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e229e8e6-9a17-4d2e-ace4-b63f82bc71aa None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:05.569184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e229e8e6-9a17-4d2e-ace4-b63f82bc71aa None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:05.569184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e229e8e6-9a17-4d2e-ace4-b63f82bc71aa None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:05.570628 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 493/990] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:08:05.575686 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54876e62-31a8-4291-972a-07fc85bfba11 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:05.576124 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54876e62-31a8-4291-972a-07fc85bfba11 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:05.576336 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54876e62-31a8-4291-972a-07fc85bfba11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:05.591676 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d623511b-8a5c-4def-8ea3-5b2b01ffe96d None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:05.591988 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d623511b-8a5c-4def-8ea3-5b2b01ffe96d None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:05.592211 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d623511b-8a5c-4def-8ea3-5b2b01ffe96d None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:05.593785 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 494/991] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:08:05.595188 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-54876e62-31a8-4291-972a-07fc85bfba11 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:05.596379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-54876e62-31a8-4291-972a-07fc85bfba11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:05.598593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a856f9bb-f390-46bb-a8d3-5de13c81dfae None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:05.599093 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a856f9bb-f390-46bb-a8d3-5de13c81dfae None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:05.599574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a856f9bb-f390-46bb-a8d3-5de13c81dfae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:05.617466 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-a856f9bb-f390-46bb-a8d3-5de13c81dfae None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:05.618662 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a856f9bb-f390-46bb-a8d3-5de13c81dfae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:05.673399 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 498/992] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:05 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:08:05.687561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 495/993] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:05 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:08:05.943018 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54876e62-31a8-4291-972a-07fc85bfba11 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:05.953399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a856f9bb-f390-46bb-a8d3-5de13c81dfae None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:05.972553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:06.034635 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:06.041914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54876e62-31a8-4291-972a-07fc85bfba11 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:06.050986 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a856f9bb-f390-46bb-a8d3-5de13c81dfae None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:06.076473 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42f0808-33f2-4766-811c-04d559a6fa98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:06.077115 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42f0808-33f2-4766-811c-04d559a6fa98 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:06.077392 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42f0808-33f2-4766-811c-04d559a6fa98 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:06.077689 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42f0808-33f2-4766-811c-04d559a6fa98 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:06.078306 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42f0808-33f2-4766-811c-04d559a6fa98 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:06.079652 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42f0808-33f2-4766-811c-04d559a6fa98 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:06.086208 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8fa193d-b5ca-49c4-8042-588246387efb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:06.086834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8fa193d-b5ca-49c4-8042-588246387efb None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:06.087109 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8fa193d-b5ca-49c4-8042-588246387efb None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:06.087376 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8fa193d-b5ca-49c4-8042-588246387efb None admin] PATH_INFO: `/v3/services/compute` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:06.087948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8fa193d-b5ca-49c4-8042-588246387efb None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:06.088215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-d42f0808-33f2-4766-811c-04d559a6fa98 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 04:08:06.088215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:06.088215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:08:06.088215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:08:06.088215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:08:06.088215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:08:06.088215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:08:06.088215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:08:06.088215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 04:08:06.088215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:06.088215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:08:06.088215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:06.088215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:06.088215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:06.088215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:06.088215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:06.088215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:06.089416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:06.089416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:06.089416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:06.089416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:06.089416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:06.089416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:06.089416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:08:06.089416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:08:06.089416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:08:06.089416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:08:06.089416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:06.089416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:06.089416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:06.089416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:06.089416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:06.089416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:06.090476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:06.090476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:06.090476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:06.090476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:06.090476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:06.090476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:06.090476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:06.090476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:06.090476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:06.090476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:06.090476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:08:06.090476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:08:06.090476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 04:08:06.090476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:06.091440 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8fa193d-b5ca-49c4-8042-588246387efb None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:06.091535 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 499/994] 10.222.0.26 () {56 vars in 1126 bytes} [Tue Sep 27 04:08:05 2022] GET /identity/v3/services/object-store => generated 93 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:08:06.091535 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7121b8db-e092-48c8-af99-f32267f9cd15 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:06.091535 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7121b8db-e092-48c8-af99-f32267f9cd15 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:06.091535 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7121b8db-e092-48c8-af99-f32267f9cd15 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:06.093787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 500/995] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:06 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:08:06.095209 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7121b8db-e092-48c8-af99-f32267f9cd15 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:06.101962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-f8fa193d-b5ca-49c4-8042-588246387efb None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 27 04:08:06.101962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:06.101962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:08:06.101962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:08:06.101962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:08:06.101962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:08:06.101962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:08:06.101962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:08:06.101962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 27 04:08:06.101962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:06.101962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:08:06.101962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:06.101962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:06.101962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:06.101962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:06.101962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:06.101962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:06.103176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:06.103176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:06.103176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:06.103176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:06.103176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:06.103176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:06.103176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:08:06.103176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:08:06.103176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:08:06.103176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:08:06.103176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:06.103176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:06.103176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:06.103176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:06.103176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:06.103176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:06.104037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:06.104037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:06.104037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:06.104037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:06.104037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:06.104037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:06.104037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:06.104037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:06.104037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:06.104037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:06.104037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:08:06.104037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:08:06.104037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 27 04:08:06.104037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:06.104037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 496/996] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:08:05 2022] GET /identity/v3/services/compute => generated 88 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:08:06.104846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df713f8b-c93d-43e4-93b1-9dd879876939 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:06.104846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df713f8b-c93d-43e4-93b1-9dd879876939 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:06.105045 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df713f8b-c93d-43e4-93b1-9dd879876939 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:06.124451 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-df713f8b-c93d-43e4-93b1-9dd879876939 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:06.125618 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-df713f8b-c93d-43e4-93b1-9dd879876939 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:06.190455 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 497/997] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:06 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:08:06.191247 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df713f8b-c93d-43e4-93b1-9dd879876939 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:06.192382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:06.203513 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7121b8db-e092-48c8-af99-f32267f9cd15 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:06.239196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11177611-e5ed-4313-bdb4-50104b92272b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:06.240358 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11177611-e5ed-4313-bdb4-50104b92272b None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:06.240625 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11177611-e5ed-4313-bdb4-50104b92272b None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:06.240899 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11177611-e5ed-4313-bdb4-50104b92272b None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:06.241520 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11177611-e5ed-4313-bdb4-50104b92272b None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:06.241856 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11177611-e5ed-4313-bdb4-50104b92272b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:06.243195 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11177611-e5ed-4313-bdb4-50104b92272b None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:06.249981 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 501/998] 10.222.0.26 () {56 vars in 1122 bytes} [Tue Sep 27 04:08:06 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:08:06.253645 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11177611-e5ed-4313-bdb4-50104b92272b None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:06.277199 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:06.301538 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df713f8b-c93d-43e4-93b1-9dd879876939 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:06.336331 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbf7d11-7b2d-423f-827c-617f60961985 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:06.336929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbf7d11-7b2d-423f-827c-617f60961985 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:06.337205 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbf7d11-7b2d-423f-827c-617f60961985 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:06.337476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbf7d11-7b2d-423f-827c-617f60961985 None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:06.338126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecbf7d11-7b2d-423f-827c-617f60961985 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:06.338425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecbf7d11-7b2d-423f-827c-617f60961985 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:06.339774 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecbf7d11-7b2d-423f-827c-617f60961985 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:06.344892 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 498/999] 10.222.0.26 () {56 vars in 1112 bytes} [Tue Sep 27 04:08:06 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:06.350908 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbf7d11-7b2d-423f-827c-617f60961985 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:06.361963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11177611-e5ed-4313-bdb4-50104b92272b None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:06.400436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc42d58-9623-497e-858d-455d264060d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:06.401304 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc42d58-9623-497e-858d-455d264060d0 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:06.401725 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc42d58-9623-497e-858d-455d264060d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:06.402236 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc42d58-9623-497e-858d-455d264060d0 None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:06.402863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc42d58-9623-497e-858d-455d264060d0 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:06.403161 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc42d58-9623-497e-858d-455d264060d0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:06.404544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc42d58-9623-497e-858d-455d264060d0 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:06.411722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 502/1000] 10.222.0.26 () {56 vars in 1122 bytes} [Tue Sep 27 04:08:06 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:08:06.417295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc42d58-9623-497e-858d-455d264060d0 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:06.444145 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:06.456874 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbf7d11-7b2d-423f-827c-617f60961985 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:06.492069 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7e26df-4fbe-413f-962f-22eeee2c5b4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:06.492659 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7e26df-4fbe-413f-962f-22eeee2c5b4b None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:06.492930 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7e26df-4fbe-413f-962f-22eeee2c5b4b None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:06.493197 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7e26df-4fbe-413f-962f-22eeee2c5b4b None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:06.493847 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7e26df-4fbe-413f-962f-22eeee2c5b4b None admin] RBAC: Adding query filter params (type=compute) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:06.494144 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7e26df-4fbe-413f-962f-22eeee2c5b4b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:06.495490 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7e26df-4fbe-413f-962f-22eeee2c5b4b None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:06.500101 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 499/1001] 10.222.0.26 () {56 vars in 1112 bytes} [Tue Sep 27 04:08:06 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:08:06.500858 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7e26df-4fbe-413f-962f-22eeee2c5b4b None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:06.509852 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:06.521718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc42d58-9623-497e-858d-455d264060d0 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:06.555445 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cc3b83-214b-48be-a5ee-a7ce1a551478 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:06.556043 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cc3b83-214b-48be-a5ee-a7ce1a551478 None admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:06.556320 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cc3b83-214b-48be-a5ee-a7ce1a551478 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:06.556597 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cc3b83-214b-48be-a5ee-a7ce1a551478 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:06.557255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6cc3b83-214b-48be-a5ee-a7ce1a551478 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:06.558636 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6cc3b83-214b-48be-a5ee-a7ce1a551478 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:06.573640 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 503/1002] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:08:06 2022] POST /identity/v3/endpoints => generated 325 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:08:06.574482 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cc3b83-214b-48be-a5ee-a7ce1a551478 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:06.605201 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7e26df-4fbe-413f-962f-22eeee2c5b4b None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:06.650061 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bda6c8c-6ccd-4a4c-9301-7d456cdcc9ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:06.650867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bda6c8c-6ccd-4a4c-9301-7d456cdcc9ec None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:06.651148 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bda6c8c-6ccd-4a4c-9301-7d456cdcc9ec None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:06.651426 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bda6c8c-6ccd-4a4c-9301-7d456cdcc9ec None admin] PATH_INFO: `/v3/services/image` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:06.652019 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bda6c8c-6ccd-4a4c-9301-7d456cdcc9ec None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:06.653454 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bda6c8c-6ccd-4a4c-9301-7d456cdcc9ec None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:06.663846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-0bda6c8c-6ccd-4a4c-9301-7d456cdcc9ec None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 27 04:08:06.663846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:06.663846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:08:06.663846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:08:06.663846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:08:06.663846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:08:06.663846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:08:06.663846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:08:06.663846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 27 04:08:06.663846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:06.663846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:08:06.663846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:06.663846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:06.663846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:06.663846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:06.663846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:06.663846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:06.665004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:06.665004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:06.665004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:06.665004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:06.665004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:06.665004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:06.665004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:08:06.665004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:08:06.665004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:08:06.665004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:08:06.665004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:06.665004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:06.665004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:06.665004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:06.665004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:06.665004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:06.665904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:06.665904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:06.665904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:06.665904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:06.665904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:06.665904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:06.665904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:06.665904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:06.665904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:06.665904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:06.665904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:08:06.665904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:08:06.665904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 27 04:08:06.665904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:06.665904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 500/1003] 10.222.0.26 () {56 vars in 1105 bytes} [Tue Sep 27 04:08:06 2022] GET /identity/v3/services/image => generated 86 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:08:06.671108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bda6c8c-6ccd-4a4c-9301-7d456cdcc9ec None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:06.672480 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:06.696261 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cc3b83-214b-48be-a5ee-a7ce1a551478 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:06.734839 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c531da-0f3d-4d3c-9dee-7135cfc7d4b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:06.735455 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c531da-0f3d-4d3c-9dee-7135cfc7d4b9 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:06.735734 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c531da-0f3d-4d3c-9dee-7135cfc7d4b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:06.736010 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c531da-0f3d-4d3c-9dee-7135cfc7d4b9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:06.736724 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c531da-0f3d-4d3c-9dee-7135cfc7d4b9 None admin] RBAC: Adding query filter params (interface=public, service_id=bb0e62d605d0426889e2d3fc1c47f0ca, region_id=RegionOne) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:06.737033 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c531da-0f3d-4d3c-9dee-7135cfc7d4b9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:06.738453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c531da-0f3d-4d3c-9dee-7135cfc7d4b9 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:06.744222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 504/1004] 10.222.0.26 () {56 vars in 1251 bytes} [Tue Sep 27 04:08:06 2022] GET /identity/v3/endpoints?service_id=bb0e62d605d0426889e2d3fc1c47f0ca&interface=public®ion_id=RegionOne => generated 194 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:06.746120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:06.750232 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c531da-0f3d-4d3c-9dee-7135cfc7d4b9 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:06.788366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bda6c8c-6ccd-4a4c-9301-7d456cdcc9ec None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:06.825157 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5f8b89-5b9e-42fb-b41c-3991849bf838 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:06.825777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5f8b89-5b9e-42fb-b41c-3991849bf838 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:06.826066 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5f8b89-5b9e-42fb-b41c-3991849bf838 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:06.826332 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5f8b89-5b9e-42fb-b41c-3991849bf838 None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:06.826951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5f8b89-5b9e-42fb-b41c-3991849bf838 None admin] RBAC: Adding query filter params (name=image) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:06.827246 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5f8b89-5b9e-42fb-b41c-3991849bf838 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:06.828582 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5f8b89-5b9e-42fb-b41c-3991849bf838 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:06.834105 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 501/1005] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:08:06 2022] GET /identity/v3/services?name=image => generated 122 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:08:06.839359 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5f8b89-5b9e-42fb-b41c-3991849bf838 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:06.866592 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c531da-0f3d-4d3c-9dee-7135cfc7d4b9 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:06.901243 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64dffc1a-5666-41fe-9c53-a0cf55e840a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:06.901885 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dffc1a-5666-41fe-9c53-a0cf55e840a1 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:06.902169 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dffc1a-5666-41fe-9c53-a0cf55e840a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:06.902446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dffc1a-5666-41fe-9c53-a0cf55e840a1 None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:06.903055 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64dffc1a-5666-41fe-9c53-a0cf55e840a1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:06.904409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64dffc1a-5666-41fe-9c53-a0cf55e840a1 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:06.910521 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 505/1006] 10.222.0.26 () {56 vars in 1087 bytes} [Tue Sep 27 04:08:06 2022] GET /identity/v3/services => generated 1727 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:08:06.952792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5f8b89-5b9e-42fb-b41c-3991849bf838 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:06.985411 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:06.992379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f190afe6-4eab-4a33-b7f9-a60862313e13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:06.993097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f190afe6-4eab-4a33-b7f9-a60862313e13 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:06.993436 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f190afe6-4eab-4a33-b7f9-a60862313e13 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:06.993805 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f190afe6-4eab-4a33-b7f9-a60862313e13 None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:06.994518 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f190afe6-4eab-4a33-b7f9-a60862313e13 None admin] RBAC: Adding query filter params (type=image) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:06.994892 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f190afe6-4eab-4a33-b7f9-a60862313e13 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:06.996342 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f190afe6-4eab-4a33-b7f9-a60862313e13 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:07.010034 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 502/1007] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:08:06 2022] GET /identity/v3/services?type=image => generated 122 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:08:08.761856 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f93c07-6116-45e5-a5a5-59ad3e4f567f None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:08.762372 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f93c07-6116-45e5-a5a5-59ad3e4f567f None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:08.762372 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f93c07-6116-45e5-a5a5-59ad3e4f567f None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:08.763735 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 506/1008] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:08:08.768916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18acf70-ac65-4650-af19-e18c5bcb94ed None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:08.769296 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18acf70-ac65-4650-af19-e18c5bcb94ed None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:08.769466 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18acf70-ac65-4650-af19-e18c5bcb94ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:08.788140 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-a18acf70-ac65-4650-af19-e18c5bcb94ed None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:08.789312 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a18acf70-ac65-4650-af19-e18c5bcb94ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:08.856773 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 503/1009] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:08 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:08:09.062968 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa66d6ef-5201-4cb5-b816-922fd408feb3 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:09.063157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa66d6ef-5201-4cb5-b816-922fd408feb3 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:09.063454 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa66d6ef-5201-4cb5-b816-922fd408feb3 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:09.065065 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 507/1010] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:08:09.071184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d042c6-f116-4a53-b011-ec3e9723c0a3 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:09.071516 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d042c6-f116-4a53-b011-ec3e9723c0a3 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:09.071743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d042c6-f116-4a53-b011-ec3e9723c0a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:09.090200 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-c7d042c6-f116-4a53-b011-ec3e9723c0a3 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:09.091372 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c7d042c6-f116-4a53-b011-ec3e9723c0a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:09.135274 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa66d6ef-5201-4cb5-b816-922fd408feb3 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:09.148104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:09.162489 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 504/1011] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:09 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:08:09.167463 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b214c2-4547-4c8a-9154-6009811de321 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:09.167751 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b214c2-4547-4c8a-9154-6009811de321 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:09.168029 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b214c2-4547-4c8a-9154-6009811de321 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:09.169398 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 505/1012] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:08:09.174039 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322c29ec-c0bb-4bc2-9de2-a19983ec1da3 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:09.174324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322c29ec-c0bb-4bc2-9de2-a19983ec1da3 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:09.174598 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322c29ec-c0bb-4bc2-9de2-a19983ec1da3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:09.192598 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-322c29ec-c0bb-4bc2-9de2-a19983ec1da3 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:09.193786 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-322c29ec-c0bb-4bc2-9de2-a19983ec1da3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:09.242879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:09.243170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa66d6ef-5201-4cb5-b816-922fd408feb3 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:09.263360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 506/1013] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:09 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 04:08:09.276721 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab54d8c8-0516-42ac-92b6-c11e0e24f55f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:09.277337 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab54d8c8-0516-42ac-92b6-c11e0e24f55f None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:09.277611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab54d8c8-0516-42ac-92b6-c11e0e24f55f None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:09.277907 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab54d8c8-0516-42ac-92b6-c11e0e24f55f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:09.278993 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab54d8c8-0516-42ac-92b6-c11e0e24f55f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:09.280397 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab54d8c8-0516-42ac-92b6-c11e0e24f55f None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:09.282071 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 508/1014] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:08:09 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:08:09.288277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322c29ec-c0bb-4bc2-9de2-a19983ec1da3 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:09.386756 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322c29ec-c0bb-4bc2-9de2-a19983ec1da3 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:09.429632 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f486b374-5cdc-400c-9b10-1ba6186a9388 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:09.430254 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f486b374-5cdc-400c-9b10-1ba6186a9388 None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:09.430527 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f486b374-5cdc-400c-9b10-1ba6186a9388 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:09.430793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f486b374-5cdc-400c-9b10-1ba6186a9388 None admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:09.431442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f486b374-5cdc-400c-9b10-1ba6186a9388 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:09.432928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f486b374-5cdc-400c-9b10-1ba6186a9388 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:09.433413 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab54d8c8-0516-42ac-92b6-c11e0e24f55f None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:09.469963 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 507/1015] 10.222.0.26 () {60 vars in 1141 bytes} [Tue Sep 27 04:08:09 2022] POST /identity/v3/projects => generated 311 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:08:09.522640 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:09.535249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f486b374-5cdc-400c-9b10-1ba6186a9388 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:09.550172 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab54d8c8-0516-42ac-92b6-c11e0e24f55f None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:09.573378 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:09.592912 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3236cab-5eb6-4d65-9759-223b326a5ef9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:09.593550 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3236cab-5eb6-4d65-9759-223b326a5ef9 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:09.593857 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3236cab-5eb6-4d65-9759-223b326a5ef9 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:09.594127 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3236cab-5eb6-4d65-9759-223b326a5ef9 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:09.594706 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3236cab-5eb6-4d65-9759-223b326a5ef9 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:09.596073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3236cab-5eb6-4d65-9759-223b326a5ef9 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:09.606213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-a3236cab-5eb6-4d65-9759-223b326a5ef9 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 27 04:08:09.606213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:09.606213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:08:09.606213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:08:09.606213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:08:09.606213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:08:09.606213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:08:09.606213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:08:09.606213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 27 04:08:09.606213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:09.606213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:08:09.606213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:09.606213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:09.606213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:09.606213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:09.606213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:09.606213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:09.607722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:09.607722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:09.607722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:09.607722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:09.607722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:09.607722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:09.607722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:08:09.607722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:08:09.607722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:08:09.607722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:08:09.607722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:09.607722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:09.607722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:09.607722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:09.607722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:09.607722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:09.608666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:09.608666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:09.608666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:09.608666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:09.608666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:09.608666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:09.608666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:09.608666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:09.608666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:09.608666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:09.608666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:08:09.608666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:08:09.608666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 27 04:08:09.608666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:09.608666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 509/1016] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:08:09 2022] GET /identity/v3/services/compute => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:08:09.614685 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3236cab-5eb6-4d65-9759-223b326a5ef9 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:09.653955 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f486b374-5cdc-400c-9b10-1ba6186a9388 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:09.695424 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea615bcd-9720-41a7-ab25-f3e2978eabed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:09.696038 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea615bcd-9720-41a7-ab25-f3e2978eabed None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:09.696316 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea615bcd-9720-41a7-ab25-f3e2978eabed None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:09.696586 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea615bcd-9720-41a7-ab25-f3e2978eabed None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:09.697239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea615bcd-9720-41a7-ab25-f3e2978eabed None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:09.698634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea615bcd-9720-41a7-ab25-f3e2978eabed None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:09.716891 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 508/1017] 10.222.0.26 () {60 vars in 1141 bytes} [Tue Sep 27 04:08:09 2022] POST /identity/v3/services => generated 242 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:08:09.735812 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3236cab-5eb6-4d65-9759-223b326a5ef9 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:09.784839 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86abdfd4-bae5-401e-a019-635d466c5c1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:09.785487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86abdfd4-bae5-401e-a019-635d466c5c1d None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:09.785723 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86abdfd4-bae5-401e-a019-635d466c5c1d None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:09.786022 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86abdfd4-bae5-401e-a019-635d466c5c1d None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:09.786700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86abdfd4-bae5-401e-a019-635d466c5c1d None admin] RBAC: Adding query filter params (name=compute) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:09.787002 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86abdfd4-bae5-401e-a019-635d466c5c1d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:09.788378 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86abdfd4-bae5-401e-a019-635d466c5c1d None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:09.794103 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 510/1018] 10.222.0.26 () {56 vars in 1112 bytes} [Tue Sep 27 04:08:09 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:09.800210 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea615bcd-9720-41a7-ab25-f3e2978eabed None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:09.903905 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:09.912838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea615bcd-9720-41a7-ab25-f3e2978eabed None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:09.948177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf76d93-1e64-4fc4-90be-8912946f3041 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:09.948794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf76d93-1e64-4fc4-90be-8912946f3041 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:09.949075 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf76d93-1e64-4fc4-90be-8912946f3041 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:09.949401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf76d93-1e64-4fc4-90be-8912946f3041 None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:09.950058 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf76d93-1e64-4fc4-90be-8912946f3041 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:09.950390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf76d93-1e64-4fc4-90be-8912946f3041 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:09.951803 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf76d93-1e64-4fc4-90be-8912946f3041 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:09.961197 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 509/1019] 10.222.0.26 () {56 vars in 1112 bytes} [Tue Sep 27 04:08:09 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:09.962479 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86abdfd4-bae5-401e-a019-635d466c5c1d None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:09.965455 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:10.064623 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86abdfd4-bae5-401e-a019-635d466c5c1d None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:10.097564 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704a411f-41bd-4594-9f19-d39f8775eb96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:10.098215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704a411f-41bd-4594-9f19-d39f8775eb96 None admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:10.098493 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704a411f-41bd-4594-9f19-d39f8775eb96 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:10.099322 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704a411f-41bd-4594-9f19-d39f8775eb96 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:10.099986 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704a411f-41bd-4594-9f19-d39f8775eb96 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:10.101325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704a411f-41bd-4594-9f19-d39f8775eb96 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:10.121583 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 511/1020] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:08:09 2022] POST /identity/v3/endpoints => generated 334 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:08:11.656230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61b22b5-40dc-4175-9a45-6c1c1cef6e6a None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:11.656737 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61b22b5-40dc-4175-9a45-6c1c1cef6e6a None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:11.656737 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61b22b5-40dc-4175-9a45-6c1c1cef6e6a None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:11.658168 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 510/1021] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:08:11.663894 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cc3dae-d458-43ee-999a-76fbef536e5e None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:11.664287 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cc3dae-d458-43ee-999a-76fbef536e5e None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:11.664471 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cc3dae-d458-43ee-999a-76fbef536e5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:11.682633 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-07cc3dae-d458-43ee-999a-76fbef536e5e None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:11.683784 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-07cc3dae-d458-43ee-999a-76fbef536e5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:11.737333 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:11.760170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 512/1022] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:11 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:08:11.921706 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e93865-d468-4dd3-8d56-c5b1aad4babb None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:11.922031 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e93865-d468-4dd3-8d56-c5b1aad4babb None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:11.922216 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e93865-d468-4dd3-8d56-c5b1aad4babb None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:11.923689 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 511/1023] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:08:11.928751 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb3444f-e14e-4a62-aba9-50f8ae1837f8 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:11.929042 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb3444f-e14e-4a62-aba9-50f8ae1837f8 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:11.929320 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb3444f-e14e-4a62-aba9-50f8ae1837f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:11.947816 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-9bb3444f-e14e-4a62-aba9-50f8ae1837f8 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:11.949007 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-9bb3444f-e14e-4a62-aba9-50f8ae1837f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:12.030089 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 513/1024] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:11 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:08:12.033399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e93865-d468-4dd3-8d56-c5b1aad4babb None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:12.183600 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e93865-d468-4dd3-8d56-c5b1aad4babb None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:12.218302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fa5705-4326-4b28-878d-9aa98cf12554 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:12.219318 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fa5705-4326-4b28-878d-9aa98cf12554 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:12.219793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fa5705-4326-4b28-878d-9aa98cf12554 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:12.220298 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fa5705-4326-4b28-878d-9aa98cf12554 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:12.221645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9fa5705-4326-4b28-878d-9aa98cf12554 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:12.223290 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9fa5705-4326-4b28-878d-9aa98cf12554 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:12.225414 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 512/1025] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:08:12 2022] GET /identity/v3/domains/default => generated 199 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:12.231061 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb3444f-e14e-4a62-aba9-50f8ae1837f8 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:12.296481 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fa5705-4326-4b28-878d-9aa98cf12554 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:12.329717 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:12.346958 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb3444f-e14e-4a62-aba9-50f8ae1837f8 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:12.362893 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:12.385344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f66401-1869-4d40-9318-b80dfd24eed3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:12.385978 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f66401-1869-4d40-9318-b80dfd24eed3 None admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:12.386260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f66401-1869-4d40-9318-b80dfd24eed3 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:12.386527 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f66401-1869-4d40-9318-b80dfd24eed3 None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:12.387211 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f66401-1869-4d40-9318-b80dfd24eed3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:12.388608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f66401-1869-4d40-9318-b80dfd24eed3 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:12.413862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 514/1026] 10.222.0.26 () {60 vars in 1132 bytes} [Tue Sep 27 04:08:12 2022] POST /identity/v3/users => generated 285 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:08:12.415114 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bea1ad-4433-4953-bd6a-1379a83dc938 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:12.415397 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bea1ad-4433-4953-bd6a-1379a83dc938 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:12.415668 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bea1ad-4433-4953-bd6a-1379a83dc938 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:12.417780 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 515/1027] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:08:12.422783 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7155640f-ce1b-4ead-b018-be7b0bbbe51c None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:12.423278 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7155640f-ce1b-4ead-b018-be7b0bbbe51c None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:12.423634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fa5705-4326-4b28-878d-9aa98cf12554 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:12.423913 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7155640f-ce1b-4ead-b018-be7b0bbbe51c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:12.445090 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-7155640f-ce1b-4ead-b018-be7b0bbbe51c None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:12.446609 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-7155640f-ce1b-4ead-b018-be7b0bbbe51c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:12.464683 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d4680e-f418-46f5-bc40-fea7162f285d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:12.465319 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d4680e-f418-46f5-bc40-fea7162f285d None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:12.465591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d4680e-f418-46f5-bc40-fea7162f285d None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:12.465902 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d4680e-f418-46f5-bc40-fea7162f285d None admin] PATH_INFO: `/v3/services/image` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:12.466494 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d4680e-f418-46f5-bc40-fea7162f285d None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:12.467878 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d4680e-f418-46f5-bc40-fea7162f285d None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:12.478727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-d0d4680e-f418-46f5-bc40-fea7162f285d None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 27 04:08:12.478727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:12.478727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:08:12.478727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:08:12.478727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:08:12.478727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:08:12.478727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:08:12.478727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:08:12.478727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 27 04:08:12.478727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:12.478727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:08:12.478727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:12.478727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:12.478727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:12.478727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:12.478727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:12.478727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:12.480038 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:12.480038 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:12.480038 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:12.480038 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:12.480038 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:12.480038 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:12.480038 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:08:12.480038 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:08:12.480038 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:08:12.480038 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:08:12.480038 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:12.480038 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:12.480038 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:12.480038 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:12.480038 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:12.480038 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:12.480879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:12.480879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:12.480879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:12.480879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:12.480879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:12.480879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:12.480879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:12.480879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:12.480879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:12.480879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:12.480879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:08:12.480879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:08:12.480879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 27 04:08:12.480879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:12.480879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 513/1028] 10.222.0.26 () {56 vars in 1105 bytes} [Tue Sep 27 04:08:12 2022] GET /identity/v3/services/image => generated 86 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:08:12.486074 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d4680e-f418-46f5-bc40-fea7162f285d None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:12.534045 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 516/1029] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:12 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:08:12.593311 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:12.601827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d4680e-f418-46f5-bc40-fea7162f285d None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:12.638980 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34136c24-4da4-4a67-ac05-21183d6fde21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:12.639821 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34136c24-4da4-4a67-ac05-21183d6fde21 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:12.640099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34136c24-4da4-4a67-ac05-21183d6fde21 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:12.640366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34136c24-4da4-4a67-ac05-21183d6fde21 None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:12.640982 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34136c24-4da4-4a67-ac05-21183d6fde21 None admin] RBAC: Adding query filter params (name=image) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:12.641278 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34136c24-4da4-4a67-ac05-21183d6fde21 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:12.642665 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34136c24-4da4-4a67-ac05-21183d6fde21 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:12.648067 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 514/1030] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:08:12 2022] GET /identity/v3/services?name=image => generated 122 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:12.653942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7155640f-ce1b-4ead-b018-be7b0bbbe51c None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:12.755483 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:12.759649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7155640f-ce1b-4ead-b018-be7b0bbbe51c None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:12.796081 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6554f0-4a84-4611-a2a7-78a5bdcacea6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:12.796681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6554f0-4a84-4611-a2a7-78a5bdcacea6 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:12.796957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6554f0-4a84-4611-a2a7-78a5bdcacea6 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:12.797224 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6554f0-4a84-4611-a2a7-78a5bdcacea6 None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:12.797872 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6554f0-4a84-4611-a2a7-78a5bdcacea6 None admin] RBAC: Adding query filter params (type=image) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:12.798177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6554f0-4a84-4611-a2a7-78a5bdcacea6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:12.799521 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6554f0-4a84-4611-a2a7-78a5bdcacea6 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:12.804879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34136c24-4da4-4a67-ac05-21183d6fde21 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:12.805340 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 517/1031] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:08:12 2022] GET /identity/v3/services?type=image => generated 350 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:12.811434 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6554f0-4a84-4611-a2a7-78a5bdcacea6 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:12.916381 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34136c24-4da4-4a67-ac05-21183d6fde21 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:12.918428 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6554f0-4a84-4611-a2a7-78a5bdcacea6 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:12.950242 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8363a7-e561-484a-93c4-cfa4a55f8ec0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:12.950866 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8363a7-e561-484a-93c4-cfa4a55f8ec0 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:12.951140 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8363a7-e561-484a-93c4-cfa4a55f8ec0 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:12.951408 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8363a7-e561-484a-93c4-cfa4a55f8ec0 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:12.954924 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0e4279-882f-4324-92cf-9de91559cbd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:12.955716 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0e4279-882f-4324-92cf-9de91559cbd5 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:12.956186 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0e4279-882f-4324-92cf-9de91559cbd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:12.956598 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0e4279-882f-4324-92cf-9de91559cbd5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:12.957452 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0e4279-882f-4324-92cf-9de91559cbd5 None admin] RBAC: Adding query filter params (interface=public, service_id=0eb3f4a275a1420e9a2856ae67bccc7b, region_id=RegionOne) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:12.958135 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0e4279-882f-4324-92cf-9de91559cbd5 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:12.959605 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0e4279-882f-4324-92cf-9de91559cbd5 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:12.961627 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8363a7-e561-484a-93c4-cfa4a55f8ec0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:08:12.966768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-3f8363a7-e561-484a-93c4-cfa4a55f8ec0 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 27 04:08:12.966768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:12.966768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:12.966768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:12.966768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:12.966768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:12.966768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:12.966768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:12.966768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:12.966768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:12.966768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:12.966768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:12.966768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:08:12.966768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:08:12.966768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:08:12.966768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:08:12.970485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:08:12.970485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:08:12.970485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:08:12.970485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:08:12.970485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:12.970485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:12.970485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:12.970485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:12.970485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:12.970485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:12.970485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:12.970485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:12.970485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:12.970485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:12.970485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:12.970485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:12.971569 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:12.971569 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:12.971569 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:12.971569 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:12.971569 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:08:12.971569 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:08:12.971569 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:08:12.971569 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:08:12.971569 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:08:12.971569 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:08:12.971569 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 27 04:08:12.971569 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:12.971569 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 515/1032] 10.222.0.26 () {56 vars in 1120 bytes} [Tue Sep 27 04:08:12 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:08:12.972466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 518/1033] 10.222.0.26 () {56 vars in 1251 bytes} [Tue Sep 27 04:08:12 2022] GET /identity/v3/endpoints?service_id=0eb3f4a275a1420e9a2856ae67bccc7b&interface=public®ion_id=RegionOne => generated 194 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:12.972566 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8363a7-e561-484a-93c4-cfa4a55f8ec0 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:12.974188 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0e4279-882f-4324-92cf-9de91559cbd5 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:12.993972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:13.007918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:13.082833 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8363a7-e561-484a-93c4-cfa4a55f8ec0 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:13.086021 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0e4279-882f-4324-92cf-9de91559cbd5 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:13.117453 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322166ef-c60f-4d2d-bb0f-448a171e5e31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:13.118073 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322166ef-c60f-4d2d-bb0f-448a171e5e31 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:13.118348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322166ef-c60f-4d2d-bb0f-448a171e5e31 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:13.118619 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322166ef-c60f-4d2d-bb0f-448a171e5e31 None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:13.119215 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322166ef-c60f-4d2d-bb0f-448a171e5e31 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:13.120572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322166ef-c60f-4d2d-bb0f-448a171e5e31 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:13.121171 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734ba216-a885-4b93-b594-18269c701ebf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:13.121910 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734ba216-a885-4b93-b594-18269c701ebf None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:13.122394 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734ba216-a885-4b93-b594-18269c701ebf None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:13.122600 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734ba216-a885-4b93-b594-18269c701ebf None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:13.123316 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734ba216-a885-4b93-b594-18269c701ebf None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:13.123681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734ba216-a885-4b93-b594-18269c701ebf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:13.125069 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734ba216-a885-4b93-b594-18269c701ebf None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:13.127585 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 516/1034] 10.222.0.26 () {56 vars in 1087 bytes} [Tue Sep 27 04:08:12 2022] GET /identity/v3/services => generated 1957 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:08:13.138048 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 519/1035] 10.222.0.26 () {56 vars in 1115 bytes} [Tue Sep 27 04:08:12 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:08:13.140315 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322166ef-c60f-4d2d-bb0f-448a171e5e31 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:13.226146 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:13.246974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322166ef-c60f-4d2d-bb0f-448a171e5e31 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:13.284424 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c542c15d-2c96-40b2-8894-fea902d7057e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:13.285125 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c542c15d-2c96-40b2-8894-fea902d7057e None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:13.285494 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c542c15d-2c96-40b2-8894-fea902d7057e None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:13.285886 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c542c15d-2c96-40b2-8894-fea902d7057e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:13.297975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c542c15d-2c96-40b2-8894-fea902d7057e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:13.299573 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c542c15d-2c96-40b2-8894-fea902d7057e None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:13.315370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-c542c15d-2c96-40b2-8894-fea902d7057e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:08:13.315370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:13.315370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:13.315370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:13.315370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:13.315370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:13.315370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:13.315370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:13.315370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:13.315370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:13.315370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:13.315370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:13.315370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:08:13.315370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:08:13.315370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:08:13.315370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:08:13.316607 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:13.316607 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:13.316607 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:13.316607 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:13.316607 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:13.316607 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:13.316607 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:13.316607 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:13.316607 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:13.316607 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:13.316607 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:13.316607 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:13.316607 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:13.316607 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:13.316607 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:13.316607 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:13.316607 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:08:13.317697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:08:13.317697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:08:13.317697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:13.322154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 517/1036] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:08:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:08:13.322629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734ba216-a885-4b93-b594-18269c701ebf None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:13.428814 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:13.439317 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734ba216-a885-4b93-b594-18269c701ebf None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:13.473944 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7646f0-da60-4212-8b56-aad982fb115b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:13.474539 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7646f0-da60-4212-8b56-aad982fb115b None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:13.474818 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7646f0-da60-4212-8b56-aad982fb115b None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:13.475091 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7646f0-da60-4212-8b56-aad982fb115b None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:13.475734 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7646f0-da60-4212-8b56-aad982fb115b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:13.476034 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7646f0-da60-4212-8b56-aad982fb115b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:13.477373 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7646f0-da60-4212-8b56-aad982fb115b None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:13.493792 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 520/1037] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:08:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:13.495338 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c542c15d-2c96-40b2-8894-fea902d7057e None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:13.597033 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c542c15d-2c96-40b2-8894-fea902d7057e None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:13.626084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:13.633942 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf39c438-d2aa-454d-af0d-8e5d089b61e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:13.634570 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf39c438-d2aa-454d-af0d-8e5d089b61e1 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:13.634842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf39c438-d2aa-454d-af0d-8e5d089b61e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:13.635163 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf39c438-d2aa-454d-af0d-8e5d089b61e1 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:13.642943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf39c438-d2aa-454d-af0d-8e5d089b61e1 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:13.644484 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf39c438-d2aa-454d-af0d-8e5d089b61e1 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:13.656167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-bf39c438-d2aa-454d-af0d-8e5d089b61e1 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 27 04:08:13.656167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:13.656167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:13.656167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:13.656167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:13.656167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:13.656167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:13.656167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:13.656167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:13.656167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:13.656167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:13.656167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:13.656167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:08:13.656167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:08:13.656167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:08:13.656167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:08:13.657161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:13.657161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:13.657161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:08:13.657161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:08:13.657161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:08:13.657161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:08:13.657161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:13.657161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:13.657161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:13.657161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:13.657161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:13.657161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:13.657161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:13.657161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:13.657161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:13.657161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:13.657161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:13.657161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:13.658152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:13.658152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:13.658152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:08:13.658152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:08:13.658152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:08:13.658152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:08:13.658152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:08:13.658152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:08:13.658152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 27 04:08:13.658152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:13.662085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 518/1038] 10.222.0.26 () {56 vars in 1093 bytes} [Tue Sep 27 04:08:13 2022] GET /identity/v3/users/nova => generated 82 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:08:13.663440 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7646f0-da60-4212-8b56-aad982fb115b None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:13.767107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7646f0-da60-4212-8b56-aad982fb115b None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:13.809727 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1a8e56-a86b-40d3-be48-c1c44631fb63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:13.810357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1a8e56-a86b-40d3-be48-c1c44631fb63 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:13.810635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1a8e56-a86b-40d3-be48-c1c44631fb63 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:13.810905 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1a8e56-a86b-40d3-be48-c1c44631fb63 None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:13.811634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e1a8e56-a86b-40d3-be48-c1c44631fb63 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:13.811940 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e1a8e56-a86b-40d3-be48-c1c44631fb63 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:13.813350 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e1a8e56-a86b-40d3-be48-c1c44631fb63 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:13.835688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 521/1039] 10.222.0.26 () {56 vars in 1133 bytes} [Tue Sep 27 04:08:13 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:13.836826 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf39c438-d2aa-454d-af0d-8e5d089b61e1 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:13.935153 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf39c438-d2aa-454d-af0d-8e5d089b61e1 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:13.967553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5fbc0c-8f54-40fa-ab1c-b525daa9320f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:13.968208 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5fbc0c-8f54-40fa-ab1c-b525daa9320f None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:13.968489 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5fbc0c-8f54-40fa-ab1c-b525daa9320f None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:13.968759 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5fbc0c-8f54-40fa-ab1c-b525daa9320f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:14.026461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:14.037255 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5fbc0c-8f54-40fa-ab1c-b525daa9320f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:14.038728 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5fbc0c-8f54-40fa-ab1c-b525daa9320f None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:14.140539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 519/1040] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:08:13 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 04:08:14.141776 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1a8e56-a86b-40d3-be48-c1c44631fb63 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:14.152492 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:14.240950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1a8e56-a86b-40d3-be48-c1c44631fb63 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:14.275214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60512652-046d-42f5-be76-48cc3ed97e75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:14.275834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60512652-046d-42f5-be76-48cc3ed97e75 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:14.276109 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60512652-046d-42f5-be76-48cc3ed97e75 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:14.276374 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60512652-046d-42f5-be76-48cc3ed97e75 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:14.286764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60512652-046d-42f5-be76-48cc3ed97e75 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:14.288179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60512652-046d-42f5-be76-48cc3ed97e75 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:14.303177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-60512652-046d-42f5-be76-48cc3ed97e75 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:08:14.303177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:14.303177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:14.303177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:14.303177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:14.303177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:14.303177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:14.303177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:14.303177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:14.303177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:14.303177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:14.303177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:14.303177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:08:14.303177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:08:14.303177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:08:14.303177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:08:14.304179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:14.304179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:14.304179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:14.304179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:14.304179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:14.304179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:14.304179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:14.304179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:14.304179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:14.304179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:14.304179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:14.304179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:14.304179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:14.304179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:14.304179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:14.304179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:14.304179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:08:14.305098 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:08:14.305098 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:08:14.305098 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:14.305098 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 522/1041] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:08:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:08:14.310991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5fbc0c-8f54-40fa-ab1c-b525daa9320f None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:14.407383 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:14.417812 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5fbc0c-8f54-40fa-ab1c-b525daa9320f None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:14.452835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec3a5ff-e6ce-47ee-9a71-e392e6476286 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:14.453439 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec3a5ff-e6ce-47ee-9a71-e392e6476286 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:14.453713 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec3a5ff-e6ce-47ee-9a71-e392e6476286 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:14.454009 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec3a5ff-e6ce-47ee-9a71-e392e6476286 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:14.454639 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec3a5ff-e6ce-47ee-9a71-e392e6476286 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:14.454933 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec3a5ff-e6ce-47ee-9a71-e392e6476286 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:14.456273 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec3a5ff-e6ce-47ee-9a71-e392e6476286 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:14.467691 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 520/1042] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:08:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:14.473088 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60512652-046d-42f5-be76-48cc3ed97e75 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:14.534294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:14.579891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60512652-046d-42f5-be76-48cc3ed97e75 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:14.614576 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7369a054-722f-439b-9566-f674356e38f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:14.615195 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7369a054-722f-439b-9566-f674356e38f3 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:14.615471 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7369a054-722f-439b-9566-f674356e38f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:14.615742 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7369a054-722f-439b-9566-f674356e38f3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:14.622240 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7369a054-722f-439b-9566-f674356e38f3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:14.623647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7369a054-722f-439b-9566-f674356e38f3 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:14.634949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-7369a054-722f-439b-9566-f674356e38f3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:08:14.634949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:14.634949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:14.634949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:14.634949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:14.634949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:14.634949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:14.634949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:14.634949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:14.634949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:14.634949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:14.634949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:14.634949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:08:14.634949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:08:14.634949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:08:14.635844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:08:14.635844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:14.635844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:14.635844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:14.635844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:14.635844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:14.635844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:14.635844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:14.635844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:14.635844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:14.635844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:14.635844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:14.635844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:14.635844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:14.635844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:14.635844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:14.635844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:14.636731 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:08:14.636731 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:08:14.636731 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:08:14.636731 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:08:14.636731 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:08:14.636731 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:08:14.636731 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:08:14.636731 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:14.636731 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 523/1043] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:08:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:08:14.643040 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec3a5ff-e6ce-47ee-9a71-e392e6476286 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:14.675228 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5baa5e0f-b37a-43d4-b0ca-fd557abef225 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:14.675511 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5baa5e0f-b37a-43d4-b0ca-fd557abef225 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:14.675850 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5baa5e0f-b37a-43d4-b0ca-fd557abef225 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:14.677234 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 524/1044] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:08:14.682137 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5d2161-a63b-477b-b2ae-ecaa3b42a5f6 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:14.682419 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5d2161-a63b-477b-b2ae-ecaa3b42a5f6 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:14.682688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5d2161-a63b-477b-b2ae-ecaa3b42a5f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:14.700397 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-ec5d2161-a63b-477b-b2ae-ecaa3b42a5f6 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:14.701559 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ec5d2161-a63b-477b-b2ae-ecaa3b42a5f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:14.745975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec3a5ff-e6ce-47ee-9a71-e392e6476286 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:14.772470 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 525/1045] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:14 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 04:08:14.782030 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee16c19-5a4e-4647-8809-91115b54315c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:14.782642 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee16c19-5a4e-4647-8809-91115b54315c None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:14.782906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee16c19-5a4e-4647-8809-91115b54315c None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:14.783175 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee16c19-5a4e-4647-8809-91115b54315c None admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:14.783857 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee16c19-5a4e-4647-8809-91115b54315c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:14.784158 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee16c19-5a4e-4647-8809-91115b54315c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:14.785620 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee16c19-5a4e-4647-8809-91115b54315c None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:14.802809 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 521/1046] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:08:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:08:14.803981 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5d2161-a63b-477b-b2ae-ecaa3b42a5f6 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:14.858190 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:14.908051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5d2161-a63b-477b-b2ae-ecaa3b42a5f6 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:14.941330 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947e8f00-a85f-4956-a7d1-adeea7de36e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:14.941956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947e8f00-a85f-4956-a7d1-adeea7de36e5 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:14.942236 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947e8f00-a85f-4956-a7d1-adeea7de36e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:14.943032 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947e8f00-a85f-4956-a7d1-adeea7de36e5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:14.943749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947e8f00-a85f-4956-a7d1-adeea7de36e5 None admin] RBAC: Adding query filter params (role.id=d25f9751bd8d4cb3963a382887a028ef, scope.project.id=c830a3bfece6456486f4b094076b5364, user.id=a1586ee841cd44658e4d5feb25de7940) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:14.944061 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947e8f00-a85f-4956-a7d1-adeea7de36e5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:14.945462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947e8f00-a85f-4956-a7d1-adeea7de36e5 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:14.958602 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 526/1047] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:08:14 2022] GET /identity/v3/role_assignments?user.id=a1586ee841cd44658e4d5feb25de7940&scope.project.id=c830a3bfece6456486f4b094076b5364&role.id=d25f9751bd8d4cb3963a382887a028ef => generated 259 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:08:15.045608 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee16c19-5a4e-4647-8809-91115b54315c None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:15.060864 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:15.153413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee16c19-5a4e-4647-8809-91115b54315c None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:15.187689 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807dd3fb-1ebf-4c51-baf8-c1daf56fc38c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:15.188341 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807dd3fb-1ebf-4c51-baf8-c1daf56fc38c None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:15.188616 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807dd3fb-1ebf-4c51-baf8-c1daf56fc38c None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:15.188885 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807dd3fb-1ebf-4c51-baf8-c1daf56fc38c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:15.201492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807dd3fb-1ebf-4c51-baf8-c1daf56fc38c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:08:15.206583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-807dd3fb-1ebf-4c51-baf8-c1daf56fc38c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:08:15.206583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:15.206583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:15.206583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:15.206583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:15.206583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:15.206583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:15.206583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:15.206583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:15.206583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:15.206583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:15.206583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:15.206583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:08:15.206583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:08:15.206583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:08:15.206583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:08:15.207705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:08:15.207705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:08:15.207705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:08:15.207705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:08:15.207705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:15.207705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:15.207705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:15.207705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:15.207705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:15.207705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:15.207705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:15.207705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:15.207705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:15.207705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:15.207705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:15.207705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:15.209184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:15.209184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:15.209184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:15.209184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:15.209184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:08:15.209184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:08:15.209184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:08:15.209184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:08:15.209184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:08:15.209184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:08:15.209184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:08:15.209184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:15.209184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 522/1048] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:08:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:08:15.214584 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947e8f00-a85f-4956-a7d1-adeea7de36e5 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:15.303016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5572a4-a451-4346-b5e0-9a4a801919d8 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:15.303191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5572a4-a451-4346-b5e0-9a4a801919d8 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:15.303483 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5572a4-a451-4346-b5e0-9a4a801919d8 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:15.305481 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 523/1049] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:08:15.310323 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6597d2c1-0cba-4574-9527-f7721d4d54fc None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:15.310601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6597d2c1-0cba-4574-9527-f7721d4d54fc None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:15.310867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6597d2c1-0cba-4574-9527-f7721d4d54fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:15.314509 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947e8f00-a85f-4956-a7d1-adeea7de36e5 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:15.329619 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-6597d2c1-0cba-4574-9527-f7721d4d54fc None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:15.329800 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:15.330673 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6597d2c1-0cba-4574-9527-f7721d4d54fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:15.352676 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395351f2-5ca2-4f05-8e6f-cedbaa0555b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:15.353279 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395351f2-5ca2-4f05-8e6f-cedbaa0555b7 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:15.353554 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395351f2-5ca2-4f05-8e6f-cedbaa0555b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:15.353855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395351f2-5ca2-4f05-8e6f-cedbaa0555b7 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:15.354505 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395351f2-5ca2-4f05-8e6f-cedbaa0555b7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:15.354802 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395351f2-5ca2-4f05-8e6f-cedbaa0555b7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:15.356141 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395351f2-5ca2-4f05-8e6f-cedbaa0555b7 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:15.365662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 527/1050] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:08:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:15.371772 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395351f2-5ca2-4f05-8e6f-cedbaa0555b7 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:15.411133 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 524/1051] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:15 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 04:08:15.477824 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395351f2-5ca2-4f05-8e6f-cedbaa0555b7 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:15.512204 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a05da4-51d9-4fb5-a279-599d9838c3e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:15.512823 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a05da4-51d9-4fb5-a279-599d9838c3e8 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:15.513096 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a05da4-51d9-4fb5-a279-599d9838c3e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:15.513363 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a05da4-51d9-4fb5-a279-599d9838c3e8 None admin] PATH_INFO: `/v3/users/181160c6cb4941d1b6f227e37b8a004d` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:15.526762 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a05da4-51d9-4fb5-a279-599d9838c3e8 None admin] RBAC: Authorizing `identity:get_user(user_id=181160c6cb4941d1b6f227e37b8a004d)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:15.527569 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a05da4-51d9-4fb5-a279-599d9838c3e8 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:15.529297 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 528/1052] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:08:15 2022] GET /identity/v3/users/181160c6cb4941d1b6f227e37b8a004d => generated 285 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:08:15.534970 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6597d2c1-0cba-4574-9527-f7721d4d54fc None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:15.537545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:15.635011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6597d2c1-0cba-4574-9527-f7721d4d54fc None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:15.668571 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487939a1-536d-4cff-bf0d-b6fd9212e594 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:15.669172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487939a1-536d-4cff-bf0d-b6fd9212e594 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:15.669445 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487939a1-536d-4cff-bf0d-b6fd9212e594 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:15.669713 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487939a1-536d-4cff-bf0d-b6fd9212e594 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:15.682232 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a05da4-51d9-4fb5-a279-599d9838c3e8 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:15.724717 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:15.743963 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487939a1-536d-4cff-bf0d-b6fd9212e594 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:15.745399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487939a1-536d-4cff-bf0d-b6fd9212e594 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:15.747225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:15.798481 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a05da4-51d9-4fb5-a279-599d9838c3e8 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:15.836474 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97aab2c5-f0ee-47aa-b7d1-ae56eb72927f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:15.837100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97aab2c5-f0ee-47aa-b7d1-ae56eb72927f None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:15.837377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97aab2c5-f0ee-47aa-b7d1-ae56eb72927f None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:15.837772 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97aab2c5-f0ee-47aa-b7d1-ae56eb72927f None admin] PATH_INFO: `/v3/services/image` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:15.838449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97aab2c5-f0ee-47aa-b7d1-ae56eb72927f None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:15.839795 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97aab2c5-f0ee-47aa-b7d1-ae56eb72927f None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:15.849486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-97aab2c5-f0ee-47aa-b7d1-ae56eb72927f None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 27 04:08:15.849486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:15.849486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:08:15.849486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:08:15.849486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:08:15.849486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:08:15.849486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:08:15.849486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:08:15.849486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 27 04:08:15.849486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:15.849486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:08:15.849486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:15.849486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:15.849486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:15.849486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:15.849486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:15.849486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:15.850496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:15.850496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:15.850496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:15.850496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:15.850496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:15.850496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:15.850496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:08:15.850496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:08:15.850496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:08:15.850496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:08:15.850496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:15.850496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:15.850496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:15.850496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:15.850496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:15.850496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:15.851327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:15.851327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:15.851327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:15.851327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:15.851327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:15.851327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:15.851327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:15.851327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:15.851327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:15.851327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:15.851327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:08:15.851327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:08:15.851327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 27 04:08:15.851327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:15.851327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 529/1053] 10.222.0.26 () {56 vars in 1105 bytes} [Tue Sep 27 04:08:15 2022] GET /identity/v3/services/image => generated 86 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:08:15.856521 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97aab2c5-f0ee-47aa-b7d1-ae56eb72927f None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:15.860242 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 525/1054] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:08:15 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 04:08:15.866419 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487939a1-536d-4cff-bf0d-b6fd9212e594 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:15.962580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:15.965878 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487939a1-536d-4cff-bf0d-b6fd9212e594 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:15.966497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97aab2c5-f0ee-47aa-b7d1-ae56eb72927f None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:15.967797 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:16.005495 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25cdfde-d274-494a-b60c-bcad06f9cd5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:16.005736 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28bf8961-6d3f-4ec6-ae99-54507d2bd413 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:16.006134 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bf8961-6d3f-4ec6-ae99-54507d2bd413 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:16.006299 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25cdfde-d274-494a-b60c-bcad06f9cd5d None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:16.006466 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25cdfde-d274-494a-b60c-bcad06f9cd5d None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:16.006609 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bf8961-6d3f-4ec6-ae99-54507d2bd413 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:16.006763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25cdfde-d274-494a-b60c-bcad06f9cd5d None admin] PATH_INFO: `/v3/projects/e84e913545a84f13889185e1422a07a2` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:16.006901 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bf8961-6d3f-4ec6-ae99-54507d2bd413 None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:16.007290 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28bf8961-6d3f-4ec6-ae99-54507d2bd413 None admin] RBAC: Adding query filter params (name=image) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:16.007591 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28bf8961-6d3f-4ec6-ae99-54507d2bd413 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:16.007749 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25cdfde-d274-494a-b60c-bcad06f9cd5d None admin] RBAC: Authorizing `identity:get_project(project_id=e84e913545a84f13889185e1422a07a2)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:16.009107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28bf8961-6d3f-4ec6-ae99-54507d2bd413 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:16.009220 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25cdfde-d274-494a-b60c-bcad06f9cd5d None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:16.011292 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 526/1055] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:08:15 2022] GET /identity/v3/projects/e84e913545a84f13889185e1422a07a2 => generated 311 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:08:16.017175 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25cdfde-d274-494a-b60c-bcad06f9cd5d None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:16.022128 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 530/1056] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:08:15 2022] GET /identity/v3/services?name=image => generated 122 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:08:16.023177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28bf8961-6d3f-4ec6-ae99-54507d2bd413 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:16.120954 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25cdfde-d274-494a-b60c-bcad06f9cd5d None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:16.131881 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28bf8961-6d3f-4ec6-ae99-54507d2bd413 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:16.156195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1626306a-45c0-4ee8-ae5d-3410a4424926 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:16.156889 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1626306a-45c0-4ee8-ae5d-3410a4424926 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:16.157169 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1626306a-45c0-4ee8-ae5d-3410a4424926 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:16.157510 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1626306a-45c0-4ee8-ae5d-3410a4424926 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:16.158199 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1626306a-45c0-4ee8-ae5d-3410a4424926 None admin] RBAC: Adding query filter params (role.id=e6a31562a18649f0809202c36a8e18ef, scope.project.id=e84e913545a84f13889185e1422a07a2, user.id=181160c6cb4941d1b6f227e37b8a004d) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:16.158526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1626306a-45c0-4ee8-ae5d-3410a4424926 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:16.159942 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1626306a-45c0-4ee8-ae5d-3410a4424926 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:16.166571 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 527/1057] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:08:16 2022] GET /identity/v3/role_assignments?user.id=181160c6cb4941d1b6f227e37b8a004d&scope.project.id=e84e913545a84f13889185e1422a07a2&role.id=e6a31562a18649f0809202c36a8e18ef => generated 259 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:08:16.175756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bcb9f9f-8fa9-400a-8d5a-fab100f5f761 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:16.176387 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcb9f9f-8fa9-400a-8d5a-fab100f5f761 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:16.176664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcb9f9f-8fa9-400a-8d5a-fab100f5f761 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:16.177002 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcb9f9f-8fa9-400a-8d5a-fab100f5f761 None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:16.177657 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bcb9f9f-8fa9-400a-8d5a-fab100f5f761 None admin] RBAC: Adding query filter params (type=image) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:16.177991 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bcb9f9f-8fa9-400a-8d5a-fab100f5f761 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:16.179416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bcb9f9f-8fa9-400a-8d5a-fab100f5f761 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:16.185793 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 531/1058] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:08:16 2022] GET /identity/v3/services?type=image => generated 350 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:16.191283 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1626306a-45c0-4ee8-ae5d-3410a4424926 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:16.230152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:16.305364 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1626306a-45c0-4ee8-ae5d-3410a4424926 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:16.341672 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3065376a-c7b9-4f1f-9b3d-e3123b27c7f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:16.342243 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3065376a-c7b9-4f1f-9b3d-e3123b27c7f0 None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:16.343132 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3065376a-c7b9-4f1f-9b3d-e3123b27c7f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:16.343132 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3065376a-c7b9-4f1f-9b3d-e3123b27c7f0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:16.343503 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3065376a-c7b9-4f1f-9b3d-e3123b27c7f0 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:16.344893 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3065376a-c7b9-4f1f-9b3d-e3123b27c7f0 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:16.367524 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 528/1059] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:08:16 2022] POST /identity/v3/endpoints => generated 327 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:08:17.104605 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561c506c-b443-4808-b635-b74c1b0dee87 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:17.105161 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561c506c-b443-4808-b635-b74c1b0dee87 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:17.105161 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561c506c-b443-4808-b635-b74c1b0dee87 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:17.106676 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 532/1060] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:08:17.111600 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372af1eb-a842-4f15-bfe8-73da7a516464 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:17.111949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372af1eb-a842-4f15-bfe8-73da7a516464 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:17.112289 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372af1eb-a842-4f15-bfe8-73da7a516464 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:17.131169 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-372af1eb-a842-4f15-bfe8-73da7a516464 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:17.132322 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-372af1eb-a842-4f15-bfe8-73da7a516464 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:17.204756 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 529/1061] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:08:17.468789 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561c506c-b443-4808-b635-b74c1b0dee87 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:17.497428 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:17.574299 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561c506c-b443-4808-b635-b74c1b0dee87 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:17.607622 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ace38a-a9a2-498c-b12c-f5d654f46886 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:17.608240 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ace38a-a9a2-498c-b12c-f5d654f46886 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:17.608515 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ace38a-a9a2-498c-b12c-f5d654f46886 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:17.608783 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ace38a-a9a2-498c-b12c-f5d654f46886 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:17.619185 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ace38a-a9a2-498c-b12c-f5d654f46886 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:08:17.623942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-f8ace38a-a9a2-498c-b12c-f5d654f46886 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 27 04:08:17.623942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:17.623942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:17.623942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:17.623942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:17.623942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:17.623942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:17.623942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:17.623942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:17.623942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:17.623942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:17.623942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:17.623942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:08:17.623942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:08:17.623942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:08:17.623942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:08:17.624862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:08:17.624862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:08:17.624862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:08:17.624862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:08:17.624862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:17.624862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:17.624862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:17.624862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:17.624862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:17.624862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:17.624862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:17.624862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:17.624862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:17.624862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:17.624862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:17.624862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:17.625702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:17.625702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:17.625702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:17.625702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:17.625702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:08:17.625702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:08:17.625702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:08:17.625702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:08:17.625702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:08:17.625702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:08:17.625702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 27 04:08:17.625702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:17.630411 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 533/1062] 10.222.0.26 () {56 vars in 1120 bytes} [Tue Sep 27 04:08:17 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:08:17.631254 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372af1eb-a842-4f15-bfe8-73da7a516464 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:17.648676 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:17.731292 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372af1eb-a842-4f15-bfe8-73da7a516464 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:17.764255 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373ff7fc-2ac8-4556-a0a8-cbd600299ebe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:17.764855 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373ff7fc-2ac8-4556-a0a8-cbd600299ebe None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:17.765128 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373ff7fc-2ac8-4556-a0a8-cbd600299ebe None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:17.765394 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373ff7fc-2ac8-4556-a0a8-cbd600299ebe None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:17.766081 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373ff7fc-2ac8-4556-a0a8-cbd600299ebe None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:17.766377 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373ff7fc-2ac8-4556-a0a8-cbd600299ebe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:17.767714 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373ff7fc-2ac8-4556-a0a8-cbd600299ebe None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:17.776211 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 530/1063] 10.222.0.26 () {56 vars in 1115 bytes} [Tue Sep 27 04:08:17 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:08:17.781706 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ace38a-a9a2-498c-b12c-f5d654f46886 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:17.875002 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:17.892454 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ace38a-a9a2-498c-b12c-f5d654f46886 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:17.925399 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57602bb-a66a-4291-b470-2c4942eed9a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:17.926062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57602bb-a66a-4291-b470-2c4942eed9a2 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:17.926343 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57602bb-a66a-4291-b470-2c4942eed9a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:17.926616 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57602bb-a66a-4291-b470-2c4942eed9a2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:17.937837 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57602bb-a66a-4291-b470-2c4942eed9a2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:17.939268 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57602bb-a66a-4291-b470-2c4942eed9a2 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:17.954979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-a57602bb-a66a-4291-b470-2c4942eed9a2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:08:17.954979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:17.954979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:17.954979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:17.954979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:17.954979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:17.954979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:17.954979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:17.954979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:17.954979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:17.954979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:17.954979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:17.954979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:08:17.954979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:08:17.954979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:08:17.954979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:08:17.955960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:17.955960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:17.955960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:17.955960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:17.955960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:17.955960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:17.955960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:17.955960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:17.955960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:17.955960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:17.955960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:17.955960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:17.955960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:17.955960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:17.955960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:17.955960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:17.955960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:08:17.961562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:08:17.961562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:08:17.961562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:17.961562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 534/1064] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:08:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:08:17.963443 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373ff7fc-2ac8-4556-a0a8-cbd600299ebe None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:18.044370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:18.068551 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373ff7fc-2ac8-4556-a0a8-cbd600299ebe None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:18.101879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51087a75-3472-46a6-9d54-c80bd01fcfb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:18.102481 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51087a75-3472-46a6-9d54-c80bd01fcfb5 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:18.102755 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51087a75-3472-46a6-9d54-c80bd01fcfb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:18.103023 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51087a75-3472-46a6-9d54-c80bd01fcfb5 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:18.103657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51087a75-3472-46a6-9d54-c80bd01fcfb5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:18.103954 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51087a75-3472-46a6-9d54-c80bd01fcfb5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:18.105297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51087a75-3472-46a6-9d54-c80bd01fcfb5 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:18.121786 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 531/1065] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:08:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:18.123230 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57602bb-a66a-4291-b470-2c4942eed9a2 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:18.229946 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57602bb-a66a-4291-b470-2c4942eed9a2 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:18.265483 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43678388-dec2-4c73-bd39-c81453d1e0da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:18.266154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43678388-dec2-4c73-bd39-c81453d1e0da None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:18.266437 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43678388-dec2-4c73-bd39-c81453d1e0da None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:18.266712 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43678388-dec2-4c73-bd39-c81453d1e0da None admin] PATH_INFO: `/v3/users/nova` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:18.274364 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43678388-dec2-4c73-bd39-c81453d1e0da None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:18.275799 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43678388-dec2-4c73-bd39-c81453d1e0da None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:18.277562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:18.289223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-43678388-dec2-4c73-bd39-c81453d1e0da None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 27 04:08:18.289223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:18.289223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:18.289223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:18.289223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:18.289223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:18.289223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:18.289223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:18.289223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:18.289223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:18.289223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:18.289223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:18.289223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:08:18.289223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:08:18.289223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:08:18.289223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:08:18.290285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:18.290285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:18.290285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:08:18.290285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:08:18.290285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:08:18.290285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:08:18.290285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:18.290285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:18.290285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:18.290285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:18.290285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:18.290285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:18.290285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:18.290285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:18.290285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:18.290285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:18.290285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:18.290285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:18.291240 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:18.291240 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:18.291240 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:08:18.291240 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:08:18.291240 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:08:18.291240 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:08:18.291240 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:08:18.291240 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:08:18.291240 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 27 04:08:18.291240 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:18.294057 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 535/1066] 10.222.0.26 () {56 vars in 1093 bytes} [Tue Sep 27 04:08:18 2022] GET /identity/v3/users/nova => generated 82 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:08:18.296086 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51087a75-3472-46a6-9d54-c80bd01fcfb5 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:18.327776 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31885d1f-5cd2-4536-a0af-3a36f416aa20 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:18.328040 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31885d1f-5cd2-4536-a0af-3a36f416aa20 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:18.328314 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31885d1f-5cd2-4536-a0af-3a36f416aa20 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:18.329776 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 536/1067] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:08:18.335022 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d040c2f6-e3f7-45f0-9d8f-dc20084e33ba None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:18.335305 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d040c2f6-e3f7-45f0-9d8f-dc20084e33ba None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:18.335578 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d040c2f6-e3f7-45f0-9d8f-dc20084e33ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:18.362003 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-d040c2f6-e3f7-45f0-9d8f-dc20084e33ba None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:18.363209 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-d040c2f6-e3f7-45f0-9d8f-dc20084e33ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:18.401159 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51087a75-3472-46a6-9d54-c80bd01fcfb5 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:18.443200 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fd73c1-ab1a-471a-9084-c221ffe853ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:18.444141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fd73c1-ab1a-471a-9084-c221ffe853ca None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:18.444565 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fd73c1-ab1a-471a-9084-c221ffe853ca None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:18.445016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fd73c1-ab1a-471a-9084-c221ffe853ca None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:18.445962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29fd73c1-ab1a-471a-9084-c221ffe853ca None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:18.446270 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29fd73c1-ab1a-471a-9084-c221ffe853ca None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:18.447748 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29fd73c1-ab1a-471a-9084-c221ffe853ca None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:18.451960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 537/1068] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:18 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 04:08:18.457359 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:18.474318 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 532/1069] 10.222.0.26 () {56 vars in 1133 bytes} [Tue Sep 27 04:08:18 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:18.476577 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d040c2f6-e3f7-45f0-9d8f-dc20084e33ba None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:18.553517 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e24c11-3c1f-42ff-8eb0-c6db4de8cd5b None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:18.553839 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e24c11-3c1f-42ff-8eb0-c6db4de8cd5b None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:18.554169 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e24c11-3c1f-42ff-8eb0-c6db4de8cd5b None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:18.555497 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 533/1070] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:08:18.560893 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcff757-a9a0-47f1-b562-85dc906c944c None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:18.561177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcff757-a9a0-47f1-b562-85dc906c944c None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:18.561448 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcff757-a9a0-47f1-b562-85dc906c944c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:18.579036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d040c2f6-e3f7-45f0-9d8f-dc20084e33ba None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:18.579478 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-adcff757-a9a0-47f1-b562-85dc906c944c None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:18.580664 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-adcff757-a9a0-47f1-b562-85dc906c944c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:18.608562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:18.617993 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f06cc0f-a1ba-46e8-984f-ae08a3682786 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:18.618624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f06cc0f-a1ba-46e8-984f-ae08a3682786 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:18.618904 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f06cc0f-a1ba-46e8-984f-ae08a3682786 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:18.619175 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f06cc0f-a1ba-46e8-984f-ae08a3682786 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:18.629568 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f06cc0f-a1ba-46e8-984f-ae08a3682786 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:18.631122 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f06cc0f-a1ba-46e8-984f-ae08a3682786 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:18.647348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-1f06cc0f-a1ba-46e8-984f-ae08a3682786 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:08:18.647348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:18.647348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:18.647348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:18.647348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:18.647348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:18.647348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:18.647348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:18.647348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:18.647348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:18.647348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:18.647348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:18.647348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:08:18.647348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:08:18.647348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:08:18.647348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:08:18.648329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:18.648329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:18.648329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:18.648329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:18.648329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:18.648329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:18.648329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:18.648329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:18.648329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:18.648329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:18.648329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:18.648329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:18.648329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:18.648329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:18.648329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:18.648329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:18.648329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:08:18.649233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:08:18.649233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:08:18.649233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:18.649233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 538/1071] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:08:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:08:18.653801 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 534/1072] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:18 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 04:08:18.654090 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f06cc0f-a1ba-46e8-984f-ae08a3682786 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:18.716719 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcff757-a9a0-47f1-b562-85dc906c944c None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:18.772246 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f06cc0f-a1ba-46e8-984f-ae08a3682786 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:18.809090 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4575e7-f4a1-4fe7-a951-310594517f23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:18.809604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4575e7-f4a1-4fe7-a951-310594517f23 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:18.809910 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4575e7-f4a1-4fe7-a951-310594517f23 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:18.810170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4575e7-f4a1-4fe7-a951-310594517f23 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:18.810797 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4575e7-f4a1-4fe7-a951-310594517f23 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:18.811087 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4575e7-f4a1-4fe7-a951-310594517f23 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:18.812423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4575e7-f4a1-4fe7-a951-310594517f23 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:18.818201 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcff757-a9a0-47f1-b562-85dc906c944c None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:18.825423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 539/1073] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:08:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:08:18.830695 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4575e7-f4a1-4fe7-a951-310594517f23 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:18.845705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:18.854979 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4151d1a-8e20-4d8a-9d36-d7177892ce54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:18.855605 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4151d1a-8e20-4d8a-9d36-d7177892ce54 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:18.855885 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4151d1a-8e20-4d8a-9d36-d7177892ce54 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:18.856154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4151d1a-8e20-4d8a-9d36-d7177892ce54 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:18.866646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4151d1a-8e20-4d8a-9d36-d7177892ce54 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:08:18.871450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-c4151d1a-8e20-4d8a-9d36-d7177892ce54 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:08:18.871450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:18.871450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:18.871450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:18.871450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:18.871450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:18.871450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:18.871450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:18.871450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:18.871450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:18.871450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:18.871450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:18.871450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:08:18.871450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:08:18.871450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:08:18.871450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:08:18.872379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:08:18.872379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:08:18.872379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:08:18.872379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:08:18.872379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:18.872379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:18.872379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:18.872379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:18.872379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:18.872379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:18.872379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:18.872379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:18.872379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:18.872379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:18.872379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:18.872379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:18.873245 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:18.873245 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:18.873245 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:18.873245 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:18.873245 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:08:18.873245 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:08:18.873245 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:08:18.873245 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:08:18.873245 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:08:18.873245 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:08:18.873245 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:08:18.873245 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:18.873245 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 535/1074] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:08:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:08:18.876324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:18.879545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4151d1a-8e20-4d8a-9d36-d7177892ce54 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:18.940277 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4575e7-f4a1-4fe7-a951-310594517f23 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:18.974264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf1b673-b65b-40b2-aa20-78cc8a3703c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:18.974885 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf1b673-b65b-40b2-aa20-78cc8a3703c1 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:18.975167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf1b673-b65b-40b2-aa20-78cc8a3703c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:18.975442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf1b673-b65b-40b2-aa20-78cc8a3703c1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:18.981117 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4151d1a-8e20-4d8a-9d36-d7177892ce54 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:18.981964 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf1b673-b65b-40b2-aa20-78cc8a3703c1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:18.983414 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf1b673-b65b-40b2-aa20-78cc8a3703c1 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:18.994497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-adf1b673-b65b-40b2-aa20-78cc8a3703c1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:08:18.994497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:18.994497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:18.994497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:18.994497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:18.994497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:18.994497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:18.994497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:18.994497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:18.994497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:18.994497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:18.994497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:18.994497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:08:18.994497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:08:18.994497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:08:18.995457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:08:18.995457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:18.995457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:18.995457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:18.995457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:18.995457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:18.995457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:18.995457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:18.995457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:18.995457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:18.995457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:18.995457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:18.995457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:18.995457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:18.995457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:18.995457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:18.995457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:18.996482 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:08:18.996482 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:08:18.996482 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:08:18.996482 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:08:18.996482 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:08:18.996482 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:08:18.996482 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:08:18.996482 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:18.996482 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 540/1075] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:08:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:08:19.001235 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf1b673-b65b-40b2-aa20-78cc8a3703c1 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:19.015861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189ddb09-e43f-435c-b941-614af5d91eb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:19.016458 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189ddb09-e43f-435c-b941-614af5d91eb5 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:19.016738 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189ddb09-e43f-435c-b941-614af5d91eb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:19.017011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189ddb09-e43f-435c-b941-614af5d91eb5 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:19.018670 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189ddb09-e43f-435c-b941-614af5d91eb5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:19.018670 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189ddb09-e43f-435c-b941-614af5d91eb5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:19.019361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189ddb09-e43f-435c-b941-614af5d91eb5 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:19.030474 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 536/1076] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:08:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:19.031381 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189ddb09-e43f-435c-b941-614af5d91eb5 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:19.080656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:19.112641 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:19.116437 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf1b673-b65b-40b2-aa20-78cc8a3703c1 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:19.131557 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189ddb09-e43f-435c-b941-614af5d91eb5 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:19.149897 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe13461c-6443-4fcd-93c2-2ce620dcacd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:19.150501 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe13461c-6443-4fcd-93c2-2ce620dcacd9 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:19.150785 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe13461c-6443-4fcd-93c2-2ce620dcacd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:19.151056 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe13461c-6443-4fcd-93c2-2ce620dcacd9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:19.164118 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6b5dc2-45b7-4085-a867-6e243c5c9bcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:19.164711 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6b5dc2-45b7-4085-a867-6e243c5c9bcd None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:19.164988 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6b5dc2-45b7-4085-a867-6e243c5c9bcd None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:19.165261 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6b5dc2-45b7-4085-a867-6e243c5c9bcd None admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:19.165971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6b5dc2-45b7-4085-a867-6e243c5c9bcd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:19.166273 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6b5dc2-45b7-4085-a867-6e243c5c9bcd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:19.167823 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6b5dc2-45b7-4085-a867-6e243c5c9bcd None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:19.179754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 537/1077] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:08:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:19.180537 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6b5dc2-45b7-4085-a867-6e243c5c9bcd None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:19.216890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe13461c-6443-4fcd-93c2-2ce620dcacd9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:19.218355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe13461c-6443-4fcd-93c2-2ce620dcacd9 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:19.279460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6b5dc2-45b7-4085-a867-6e243c5c9bcd None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:19.288300 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:19.312192 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:19.313707 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3838dabd-82a3-47cb-91e7-bbe4f8619b89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:19.314370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3838dabd-82a3-47cb-91e7-bbe4f8619b89 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:19.314650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3838dabd-82a3-47cb-91e7-bbe4f8619b89 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:19.314925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3838dabd-82a3-47cb-91e7-bbe4f8619b89 None admin] PATH_INFO: `/v3/users/181160c6cb4941d1b6f227e37b8a004d` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:19.316082 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3838dabd-82a3-47cb-91e7-bbe4f8619b89 None admin] RBAC: Authorizing `identity:get_user(user_id=181160c6cb4941d1b6f227e37b8a004d)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:19.318093 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 541/1078] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:08:18 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 04:08:19.318224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3838dabd-82a3-47cb-91e7-bbe4f8619b89 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:19.319158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe13461c-6443-4fcd-93c2-2ce620dcacd9 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:19.319690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 538/1079] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:08:19 2022] GET /identity/v3/users/181160c6cb4941d1b6f227e37b8a004d => generated 285 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:19.323654 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3838dabd-82a3-47cb-91e7-bbe4f8619b89 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:19.419171 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe13461c-6443-4fcd-93c2-2ce620dcacd9 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:19.422381 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3838dabd-82a3-47cb-91e7-bbe4f8619b89 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:19.453145 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c1f01b-f3b1-40a4-a728-5b18df6217f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:19.453830 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c1f01b-f3b1-40a4-a728-5b18df6217f9 None admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:19.454112 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c1f01b-f3b1-40a4-a728-5b18df6217f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:19.454388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c1f01b-f3b1-40a4-a728-5b18df6217f9 None admin] PATH_INFO: `/v3/projects/c830a3bfece6456486f4b094076b5364/users/a1586ee841cd44658e4d5feb25de7940/roles/d25f9751bd8d4cb3963a382887a028ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:19.456429 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271a80b8-d63d-465c-92b2-bb8faa403f66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:19.456562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c1f01b-f3b1-40a4-a728-5b18df6217f9 None admin] RBAC: Authorizing `identity:create_grant(project_id=c830a3bfece6456486f4b094076b5364, user_id=a1586ee841cd44658e4d5feb25de7940, role_id=d25f9751bd8d4cb3963a382887a028ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:19.457097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271a80b8-d63d-465c-92b2-bb8faa403f66 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:19.457374 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271a80b8-d63d-465c-92b2-bb8faa403f66 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:19.457645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271a80b8-d63d-465c-92b2-bb8faa403f66 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:19.457989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c1f01b-f3b1-40a4-a728-5b18df6217f9 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:19.468456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271a80b8-d63d-465c-92b2-bb8faa403f66 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:19.469912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271a80b8-d63d-465c-92b2-bb8faa403f66 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:19.471722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 542/1080] 10.222.0.26 () {58 vars in 1439 bytes} [Tue Sep 27 04:08:19 2022] PUT /identity/v3/projects/c830a3bfece6456486f4b094076b5364/users/a1586ee841cd44658e4d5feb25de7940/roles/d25f9751bd8d4cb3963a382887a028ef => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:08:19.472510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c1f01b-f3b1-40a4-a728-5b18df6217f9 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:19.490903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-271a80b8-d63d-465c-92b2-bb8faa403f66 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:08:19.490903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:19.490903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:19.490903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:19.490903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:19.490903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:19.490903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:19.490903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:19.490903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:19.490903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:19.490903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:19.490903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:19.490903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:08:19.490903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:08:19.490903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:08:19.490903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:08:19.492011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:19.492011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:19.492011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:19.492011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:19.492011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:19.492011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:19.492011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:19.492011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:19.492011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:19.492011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:19.492011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:19.492011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:19.492011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:19.492011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:19.492011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:19.492011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:19.492011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:08:19.492932 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:08:19.492932 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:08:19.492932 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:19.492932 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 539/1081] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:08:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:08:19.498089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271a80b8-d63d-465c-92b2-bb8faa403f66 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:19.530329 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:19.554713 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:19.607512 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c1f01b-f3b1-40a4-a728-5b18df6217f9 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:19.615844 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271a80b8-d63d-465c-92b2-bb8faa403f66 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:19.646878 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce24e93-aabe-4778-ab25-6c19ac2099ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:19.647632 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce24e93-aabe-4778-ab25-6c19ac2099ff None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:19.647909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce24e93-aabe-4778-ab25-6c19ac2099ff None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:19.648177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce24e93-aabe-4778-ab25-6c19ac2099ff None admin] PATH_INFO: `/v3/projects/e84e913545a84f13889185e1422a07a2` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:19.649298 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce24e93-aabe-4778-ab25-6c19ac2099ff None admin] RBAC: Authorizing `identity:get_project(project_id=e84e913545a84f13889185e1422a07a2)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:19.651317 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce24e93-aabe-4778-ab25-6c19ac2099ff None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:19.652666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 543/1082] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:08:19 2022] GET /identity/v3/projects/e84e913545a84f13889185e1422a07a2 => generated 311 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:19.659810 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae38b54a-cecf-4405-b7b6-46b8a34bcca0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:19.659983 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce24e93-aabe-4778-ab25-6c19ac2099ff None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:19.660831 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae38b54a-cecf-4405-b7b6-46b8a34bcca0 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:19.661958 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae38b54a-cecf-4405-b7b6-46b8a34bcca0 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:19.662467 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae38b54a-cecf-4405-b7b6-46b8a34bcca0 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:19.663315 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae38b54a-cecf-4405-b7b6-46b8a34bcca0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:19.663825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae38b54a-cecf-4405-b7b6-46b8a34bcca0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:19.665398 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae38b54a-cecf-4405-b7b6-46b8a34bcca0 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:19.679241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 540/1083] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:08:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:08:19.769778 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce24e93-aabe-4778-ab25-6c19ac2099ff None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:19.798611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:19.808172 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0f862b-8aa7-48ec-b6f6-b2a935245854 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:19.809254 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0f862b-8aa7-48ec-b6f6-b2a935245854 None admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:19.809796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0f862b-8aa7-48ec-b6f6-b2a935245854 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:19.810249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0f862b-8aa7-48ec-b6f6-b2a935245854 None admin] PATH_INFO: `/v3/projects/e84e913545a84f13889185e1422a07a2/users/181160c6cb4941d1b6f227e37b8a004d/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:19.812445 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0f862b-8aa7-48ec-b6f6-b2a935245854 None admin] RBAC: Authorizing `identity:create_grant(project_id=e84e913545a84f13889185e1422a07a2, user_id=181160c6cb4941d1b6f227e37b8a004d, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:19.814160 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0f862b-8aa7-48ec-b6f6-b2a935245854 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:19.831040 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 544/1084] 10.222.0.26 () {58 vars in 1439 bytes} [Tue Sep 27 04:08:19 2022] PUT /identity/v3/projects/e84e913545a84f13889185e1422a07a2/users/181160c6cb4941d1b6f227e37b8a004d/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:08:21.647304 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec365469-ae54-4c8a-b4d4-1c88d1e60a82 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:21.647824 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec365469-ae54-4c8a-b4d4-1c88d1e60a82 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:21.647824 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec365469-ae54-4c8a-b4d4-1c88d1e60a82 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:21.649162 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 541/1085] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:08:21.654520 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe76d6d9-6f63-4a00-8446-72e6c05813e3 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:21.654941 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe76d6d9-6f63-4a00-8446-72e6c05813e3 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:21.655048 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe76d6d9-6f63-4a00-8446-72e6c05813e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:21.673446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-fe76d6d9-6f63-4a00-8446-72e6c05813e3 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:21.674571 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-fe76d6d9-6f63-4a00-8446-72e6c05813e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:21.738179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 545/1086] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:08:21.978492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d688942-ab98-4293-8026-02c69c6d4bf1 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:21.978781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d688942-ab98-4293-8026-02c69c6d4bf1 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:21.979058 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d688942-ab98-4293-8026-02c69c6d4bf1 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:21.980381 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 542/1087] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:08:21.985731 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced773b5-a257-4279-a9ce-b15d53344c2e None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:21.986043 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced773b5-a257-4279-a9ce-b15d53344c2e None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:21.986318 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced773b5-a257-4279-a9ce-b15d53344c2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:22.004052 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-ced773b5-a257-4279-a9ce-b15d53344c2e None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:22.004390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d688942-ab98-4293-8026-02c69c6d4bf1 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:22.005247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ced773b5-a257-4279-a9ce-b15d53344c2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:22.073221 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 546/1088] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:08:22.095032 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:22.108085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d688942-ab98-4293-8026-02c69c6d4bf1 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:22.140416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6efd604-b247-4595-accf-fff462bc3f5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:22.141029 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6efd604-b247-4595-accf-fff462bc3f5e None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:22.141302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6efd604-b247-4595-accf-fff462bc3f5e None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:22.141567 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6efd604-b247-4595-accf-fff462bc3f5e None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:22.151756 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6efd604-b247-4595-accf-fff462bc3f5e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:08:22.156529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-f6efd604-b247-4595-accf-fff462bc3f5e None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 27 04:08:22.156529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:22.156529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:22.156529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:22.156529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:22.156529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:22.156529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:22.156529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:22.156529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:22.156529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:22.156529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:22.156529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:22.156529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:08:22.156529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:08:22.156529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:08:22.156529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:08:22.157447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:08:22.157447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:08:22.157447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:08:22.157447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:08:22.157447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:22.157447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:22.157447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:22.157447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:22.157447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:22.157447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:22.157447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:22.157447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:22.157447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:22.157447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:22.157447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:22.157447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:22.158468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:22.158468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:22.158468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:22.158468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:22.158468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:08:22.158468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:08:22.158468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:08:22.158468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:08:22.158468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:08:22.158468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:08:22.158468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 27 04:08:22.158468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:22.158468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 543/1089] 10.222.0.26 () {56 vars in 1120 bytes} [Tue Sep 27 04:08:22 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:08:22.163658 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced773b5-a257-4279-a9ce-b15d53344c2e None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:22.211033 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:22.263079 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced773b5-a257-4279-a9ce-b15d53344c2e None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:22.297314 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ad5245-0e63-49c5-8eb6-9a2a6d9f9e49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:22.297945 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ad5245-0e63-49c5-8eb6-9a2a6d9f9e49 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:22.298228 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ad5245-0e63-49c5-8eb6-9a2a6d9f9e49 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:22.298503 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ad5245-0e63-49c5-8eb6-9a2a6d9f9e49 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:22.299181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ad5245-0e63-49c5-8eb6-9a2a6d9f9e49 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:22.299469 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ad5245-0e63-49c5-8eb6-9a2a6d9f9e49 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:22.300816 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ad5245-0e63-49c5-8eb6-9a2a6d9f9e49 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:22.308763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 547/1090] 10.222.0.26 () {56 vars in 1115 bytes} [Tue Sep 27 04:08:22 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:22.314737 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6efd604-b247-4595-accf-fff462bc3f5e None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:22.342710 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ad5245-0e63-49c5-8eb6-9a2a6d9f9e49 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:22.413213 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6efd604-b247-4595-accf-fff462bc3f5e None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:22.442839 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ad5245-0e63-49c5-8eb6-9a2a6d9f9e49 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:22.447004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564a072b-c824-47f9-b24d-0da0eb8921a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:22.447633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564a072b-c824-47f9-b24d-0da0eb8921a6 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:22.447911 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564a072b-c824-47f9-b24d-0da0eb8921a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:22.448184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564a072b-c824-47f9-b24d-0da0eb8921a6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:22.452785 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:22.458670 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564a072b-c824-47f9-b24d-0da0eb8921a6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:22.460096 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564a072b-c824-47f9-b24d-0da0eb8921a6 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:22.468289 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:22.474820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-564a072b-c824-47f9-b24d-0da0eb8921a6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:08:22.474820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:22.474820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:22.474820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:22.474820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:22.474820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:22.474820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:22.474820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:22.474820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:22.474820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:22.474820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:22.474820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:22.474820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:08:22.474820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:08:22.474820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:08:22.474820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:08:22.475972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:22.475972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:22.475972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:22.475972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:22.475972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:22.475972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:22.475972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:22.475972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:22.475972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:22.475972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:22.475972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:22.475972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:22.475972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:22.475972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:22.475972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:22.475972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:22.475972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:08:22.477100 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:08:22.477100 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:08:22.477100 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:22.477100 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 544/1091] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:08:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:08:22.479520 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fe54aa-7836-4f4a-9167-72e7256a96c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:22.480134 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fe54aa-7836-4f4a-9167-72e7256a96c1 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:22.480414 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fe54aa-7836-4f4a-9167-72e7256a96c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:22.480688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fe54aa-7836-4f4a-9167-72e7256a96c1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:22.483392 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564a072b-c824-47f9-b24d-0da0eb8921a6 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:22.493293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fe54aa-7836-4f4a-9167-72e7256a96c1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:08:22.498094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-c9fe54aa-7836-4f4a-9167-72e7256a96c1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:08:22.498094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:22.498094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:22.498094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:22.498094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:22.498094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:22.498094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:22.498094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:22.498094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:22.498094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:22.498094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:22.498094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:22.498094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:08:22.498094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:08:22.498094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:08:22.498094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:08:22.499027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:08:22.499027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:08:22.499027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:08:22.499027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:08:22.499027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:22.499027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:22.499027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:22.499027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:22.499027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:22.499027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:22.499027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:22.499027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:22.499027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:22.499027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:22.499027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:22.499027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:22.499874 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:22.499874 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:22.499874 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:22.499874 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:22.499874 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:08:22.499874 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:08:22.499874 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:08:22.499874 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:08:22.499874 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:08:22.499874 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:08:22.499874 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:08:22.499874 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:22.499874 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 548/1092] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:08:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:08:22.506024 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fe54aa-7836-4f4a-9167-72e7256a96c1 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:22.579985 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564a072b-c824-47f9-b24d-0da0eb8921a6 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:22.604963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fe54aa-7836-4f4a-9167-72e7256a96c1 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:22.612405 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f60c013-e18e-4bc4-a733-b00f79bd3c35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:22.613030 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f60c013-e18e-4bc4-a733-b00f79bd3c35 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:22.613302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f60c013-e18e-4bc4-a733-b00f79bd3c35 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:22.613565 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f60c013-e18e-4bc4-a733-b00f79bd3c35 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:22.614932 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f60c013-e18e-4bc4-a733-b00f79bd3c35 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:22.615224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f60c013-e18e-4bc4-a733-b00f79bd3c35 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:22.616568 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f60c013-e18e-4bc4-a733-b00f79bd3c35 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:22.628645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 545/1093] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:08:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:08:22.635063 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f60c013-e18e-4bc4-a733-b00f79bd3c35 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:22.646205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0457817-67d5-4d2b-9681-d84afe95c2d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:22.646806 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0457817-67d5-4d2b-9681-d84afe95c2d7 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:22.647091 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0457817-67d5-4d2b-9681-d84afe95c2d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:22.647363 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0457817-67d5-4d2b-9681-d84afe95c2d7 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:22.648016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0457817-67d5-4d2b-9681-d84afe95c2d7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:22.648319 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0457817-67d5-4d2b-9681-d84afe95c2d7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:22.649669 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0457817-67d5-4d2b-9681-d84afe95c2d7 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:22.658140 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 549/1094] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:08:22 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:08:22.664089 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0457817-67d5-4d2b-9681-d84afe95c2d7 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:22.694105 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:22.698909 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:22.737409 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f60c013-e18e-4bc4-a733-b00f79bd3c35 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:22.764733 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0457817-67d5-4d2b-9681-d84afe95c2d7 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:22.770603 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9f0988-8c11-4d8a-a5a8-93edb5a8ed23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:22.771230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9f0988-8c11-4d8a-a5a8-93edb5a8ed23 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:22.771514 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9f0988-8c11-4d8a-a5a8-93edb5a8ed23 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:22.771789 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9f0988-8c11-4d8a-a5a8-93edb5a8ed23 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:22.778586 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9f0988-8c11-4d8a-a5a8-93edb5a8ed23 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:22.780021 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9f0988-8c11-4d8a-a5a8-93edb5a8ed23 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:22.791602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-dc9f0988-8c11-4d8a-a5a8-93edb5a8ed23 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 27 04:08:22.791602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:22.791602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:22.791602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:22.791602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:22.791602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:22.791602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:22.791602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:22.791602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:22.791602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:22.791602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:22.791602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:22.791602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:08:22.791602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:08:22.791602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:08:22.791602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:08:22.792561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:22.792561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:22.792561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:08:22.792561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:08:22.792561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:08:22.792561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:08:22.792561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:22.792561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:22.792561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:22.792561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:22.792561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:22.792561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:22.792561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:22.792561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:22.792561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:22.792561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:22.792561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:22.792561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:22.793512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:22.793512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:22.793512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:08:22.793512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:08:22.793512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:08:22.793512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:08:22.793512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:08:22.793512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:08:22.793512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 27 04:08:22.793512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:22.793512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 546/1095] 10.222.0.26 () {56 vars in 1093 bytes} [Tue Sep 27 04:08:22 2022] GET /identity/v3/users/nova => generated 82 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:08:22.797146 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165c5e7e-69e1-4375-8ad7-51feba29d1b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:22.797981 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165c5e7e-69e1-4375-8ad7-51feba29d1b8 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:22.798266 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165c5e7e-69e1-4375-8ad7-51feba29d1b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:22.798468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9f0988-8c11-4d8a-a5a8-93edb5a8ed23 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:22.798552 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165c5e7e-69e1-4375-8ad7-51feba29d1b8 None admin] PATH_INFO: `/v3/users/181160c6cb4941d1b6f227e37b8a004d` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:22.799634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165c5e7e-69e1-4375-8ad7-51feba29d1b8 None admin] RBAC: Authorizing `identity:get_user(user_id=181160c6cb4941d1b6f227e37b8a004d)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:22.801043 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165c5e7e-69e1-4375-8ad7-51feba29d1b8 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:22.807684 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 550/1096] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:08:22 2022] GET /identity/v3/users/181160c6cb4941d1b6f227e37b8a004d => generated 285 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:08:22.808591 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165c5e7e-69e1-4375-8ad7-51feba29d1b8 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:22.897128 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9f0988-8c11-4d8a-a5a8-93edb5a8ed23 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:22.902188 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:22.905237 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165c5e7e-69e1-4375-8ad7-51feba29d1b8 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:22.916609 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:22.931851 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84fd00cc-9fc6-43f5-b704-10d2a7a25426 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:22.932446 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fd00cc-9fc6-43f5-b704-10d2a7a25426 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:22.932719 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fd00cc-9fc6-43f5-b704-10d2a7a25426 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:22.932985 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fd00cc-9fc6-43f5-b704-10d2a7a25426 None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:22.933698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84fd00cc-9fc6-43f5-b704-10d2a7a25426 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:22.934019 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84fd00cc-9fc6-43f5-b704-10d2a7a25426 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:22.935422 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84fd00cc-9fc6-43f5-b704-10d2a7a25426 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:22.939780 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd95180a-6133-48f1-8372-68a89f19f5c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:22.940375 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd95180a-6133-48f1-8372-68a89f19f5c8 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:22.940657 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd95180a-6133-48f1-8372-68a89f19f5c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:22.940933 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd95180a-6133-48f1-8372-68a89f19f5c8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:22.952472 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 547/1097] 10.222.0.26 () {56 vars in 1133 bytes} [Tue Sep 27 04:08:22 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:08:22.958164 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84fd00cc-9fc6-43f5-b704-10d2a7a25426 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:23.007423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd95180a-6133-48f1-8372-68a89f19f5c8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:23.008847 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd95180a-6133-48f1-8372-68a89f19f5c8 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:23.054324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84fd00cc-9fc6-43f5-b704-10d2a7a25426 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:23.087414 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc2835f-877f-4ec9-9283-ed38af07795a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:23.088004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc2835f-877f-4ec9-9283-ed38af07795a None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:23.088284 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc2835f-877f-4ec9-9283-ed38af07795a None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:23.088555 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc2835f-877f-4ec9-9283-ed38af07795a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:23.100945 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:23.109627 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 551/1098] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:08:22 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 04:08:23.111018 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd95180a-6133-48f1-8372-68a89f19f5c8 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:23.133825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:23.154904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc2835f-877f-4ec9-9283-ed38af07795a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:23.156265 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc2835f-877f-4ec9-9283-ed38af07795a None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:23.205987 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd95180a-6133-48f1-8372-68a89f19f5c8 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:23.241333 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f446fa68-945d-4241-9831-4b6c9a3fba4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:23.241977 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f446fa68-945d-4241-9831-4b6c9a3fba4f None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:23.242251 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f446fa68-945d-4241-9831-4b6c9a3fba4f None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:23.242525 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f446fa68-945d-4241-9831-4b6c9a3fba4f None admin] PATH_INFO: `/v3/projects/e84e913545a84f13889185e1422a07a2` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:23.243608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f446fa68-945d-4241-9831-4b6c9a3fba4f None admin] RBAC: Authorizing `identity:get_project(project_id=e84e913545a84f13889185e1422a07a2)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:23.244991 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f446fa68-945d-4241-9831-4b6c9a3fba4f None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:23.246610 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 552/1099] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:08:23 2022] GET /identity/v3/projects/e84e913545a84f13889185e1422a07a2 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:23.252531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f446fa68-945d-4241-9831-4b6c9a3fba4f None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:23.262836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 548/1100] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:08:22 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 27 04:08:23.263853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc2835f-877f-4ec9-9283-ed38af07795a None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:23.316721 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:23.338671 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:23.349793 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f446fa68-945d-4241-9831-4b6c9a3fba4f None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:23.359479 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc2835f-877f-4ec9-9283-ed38af07795a None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:23.383897 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64735f27-a8ec-47c4-a268-51fbdc06b8ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:23.384709 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64735f27-a8ec-47c4-a268-51fbdc06b8ee None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:23.384988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64735f27-a8ec-47c4-a268-51fbdc06b8ee None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:23.385252 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64735f27-a8ec-47c4-a268-51fbdc06b8ee None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:23.385989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64735f27-a8ec-47c4-a268-51fbdc06b8ee None admin] RBAC: Adding query filter params (role.id=e6a31562a18649f0809202c36a8e18ef, scope.project.id=e84e913545a84f13889185e1422a07a2, user.id=181160c6cb4941d1b6f227e37b8a004d) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:23.386291 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64735f27-a8ec-47c4-a268-51fbdc06b8ee None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:23.387687 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64735f27-a8ec-47c4-a268-51fbdc06b8ee None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:23.393504 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 553/1101] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:08:23 2022] GET /identity/v3/role_assignments?user.id=181160c6cb4941d1b6f227e37b8a004d&scope.project.id=e84e913545a84f13889185e1422a07a2&role.id=e6a31562a18649f0809202c36a8e18ef => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:23.394080 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a13f788-9121-4590-9e03-222e68ee38ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:23.394700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a13f788-9121-4590-9e03-222e68ee38ae None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:23.394978 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a13f788-9121-4590-9e03-222e68ee38ae None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:23.395249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a13f788-9121-4590-9e03-222e68ee38ae None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:23.406206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a13f788-9121-4590-9e03-222e68ee38ae None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:23.407584 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a13f788-9121-4590-9e03-222e68ee38ae None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:23.424071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-4a13f788-9121-4590-9e03-222e68ee38ae None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:08:23.424071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:23.424071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:23.424071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:23.424071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:23.424071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:23.424071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:23.424071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:23.424071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:23.424071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:23.424071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:23.424071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:23.424071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:08:23.424071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:08:23.424071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:08:23.424071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:08:23.425313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:23.425313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:23.425313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:23.425313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:23.425313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:23.425313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:23.425313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:23.425313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:23.425313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:23.425313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:23.425313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:23.425313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:23.425313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:23.425313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:23.425313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:23.425313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:23.425313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:08:23.426275 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:08:23.426275 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:08:23.426275 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:23.426275 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 549/1102] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:08:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:08:23.431904 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64735f27-a8ec-47c4-a268-51fbdc06b8ee None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:23.533911 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64735f27-a8ec-47c4-a268-51fbdc06b8ee None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:23.563221 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:23.569181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb6ea77-712b-4ff3-b572-57ec8a77c4c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:23.569803 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb6ea77-712b-4ff3-b572-57ec8a77c4c9 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:23.570080 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb6ea77-712b-4ff3-b572-57ec8a77c4c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:23.570348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb6ea77-712b-4ff3-b572-57ec8a77c4c9 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:23.571059 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb6ea77-712b-4ff3-b572-57ec8a77c4c9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:23.571313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb6ea77-712b-4ff3-b572-57ec8a77c4c9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:23.572647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb6ea77-712b-4ff3-b572-57ec8a77c4c9 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:23.584519 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 554/1103] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:08:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:23.590500 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a13f788-9121-4590-9e03-222e68ee38ae None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:23.689596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a13f788-9121-4590-9e03-222e68ee38ae None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:23.715857 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:23.721917 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a76f8e-1438-4520-806b-7035311f9509 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:23.722533 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a76f8e-1438-4520-806b-7035311f9509 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:23.722804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a76f8e-1438-4520-806b-7035311f9509 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:23.723068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a76f8e-1438-4520-806b-7035311f9509 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:23.729968 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a76f8e-1438-4520-806b-7035311f9509 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:23.731618 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a76f8e-1438-4520-806b-7035311f9509 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:23.743099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-c2a76f8e-1438-4520-806b-7035311f9509 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:08:23.743099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:23.743099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:23.743099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:23.743099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:23.743099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:23.743099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:23.743099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:23.743099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:23.743099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:23.743099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:23.743099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:23.743099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:08:23.743099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:08:23.743099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:08:23.744137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:08:23.744137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:23.744137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:23.744137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:23.744137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:23.744137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:23.744137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:23.744137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:23.744137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:23.744137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:23.744137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:23.744137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:23.744137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:23.744137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:23.744137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:23.744137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:23.744137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:23.745035 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:08:23.745035 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:08:23.745035 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:08:23.745035 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:08:23.745035 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:08:23.745035 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:08:23.745035 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:08:23.745035 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:23.745679 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 550/1104] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:08:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:08:23.750280 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb6ea77-712b-4ff3-b572-57ec8a77c4c9 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:23.850551 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb6ea77-712b-4ff3-b572-57ec8a77c4c9 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:23.883148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd2bcb4-6498-442d-bbc0-3817c9209a42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:23.883738 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd2bcb4-6498-442d-bbc0-3817c9209a42 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:23.884015 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd2bcb4-6498-442d-bbc0-3817c9209a42 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:23.884283 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd2bcb4-6498-442d-bbc0-3817c9209a42 None admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:23.884962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd2bcb4-6498-442d-bbc0-3817c9209a42 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:23.885260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd2bcb4-6498-442d-bbc0-3817c9209a42 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:23.886689 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd2bcb4-6498-442d-bbc0-3817c9209a42 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:23.899198 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 555/1105] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:08:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:23.905180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a76f8e-1438-4520-806b-7035311f9509 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:24.001425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a76f8e-1438-4520-806b-7035311f9509 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:24.033838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3bbf748-1c88-4029-91e9-2819a5ffb94e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:24.034424 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bbf748-1c88-4029-91e9-2819a5ffb94e None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:24.034695 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bbf748-1c88-4029-91e9-2819a5ffb94e None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:24.034960 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bbf748-1c88-4029-91e9-2819a5ffb94e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:24.035669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3bbf748-1c88-4029-91e9-2819a5ffb94e None admin] RBAC: Adding query filter params (role.id=d25f9751bd8d4cb3963a382887a028ef, scope.project.id=c830a3bfece6456486f4b094076b5364, user.id=a1586ee841cd44658e4d5feb25de7940) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:24.035970 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3bbf748-1c88-4029-91e9-2819a5ffb94e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:24.037385 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3bbf748-1c88-4029-91e9-2819a5ffb94e None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:24.042919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 551/1106] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:08:23 2022] GET /identity/v3/role_assignments?user.id=a1586ee841cd44658e4d5feb25de7940&scope.project.id=c830a3bfece6456486f4b094076b5364&role.id=d25f9751bd8d4cb3963a382887a028ef => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:08:25.571949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da77f7e-d9ee-4ea1-8116-e3d192fe5792 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:25.572510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da77f7e-d9ee-4ea1-8116-e3d192fe5792 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:25.572510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da77f7e-d9ee-4ea1-8116-e3d192fe5792 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:25.573817 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 556/1107] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:08:25.578883 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8588a5-13c6-460a-8088-ead4e590e6be None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:25.579325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8588a5-13c6-460a-8088-ead4e590e6be None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:25.579424 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8588a5-13c6-460a-8088-ead4e590e6be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:25.597371 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-3e8588a5-13c6-460a-8088-ead4e590e6be None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:25.598563 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-3e8588a5-13c6-460a-8088-ead4e590e6be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:25.644739 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:25.662216 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 552/1108] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:25 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:08:25.930317 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da77f7e-d9ee-4ea1-8116-e3d192fe5792 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:25.982740 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:26.031889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da77f7e-d9ee-4ea1-8116-e3d192fe5792 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:26.067255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6213cb3-4376-497d-b46a-f6893d027589 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:26.068421 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6213cb3-4376-497d-b46a-f6893d027589 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:26.068700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6213cb3-4376-497d-b46a-f6893d027589 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:26.069024 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6213cb3-4376-497d-b46a-f6893d027589 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:26.070464 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6213cb3-4376-497d-b46a-f6893d027589 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:26.072616 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6213cb3-4376-497d-b46a-f6893d027589 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:26.074860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 557/1109] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:08:25 2022] GET /identity/v3/domains/default => generated 199 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:26.080651 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e8588a5-13c6-460a-8088-ead4e590e6be None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:26.173953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e8588a5-13c6-460a-8088-ead4e590e6be None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:26.205884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362eaee3-d79c-4cf0-a481-25d60bc67c23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:26.206399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362eaee3-d79c-4cf0-a481-25d60bc67c23 None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:26.206671 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362eaee3-d79c-4cf0-a481-25d60bc67c23 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:26.206935 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362eaee3-d79c-4cf0-a481-25d60bc67c23 None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:26.207574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362eaee3-d79c-4cf0-a481-25d60bc67c23 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:26.209524 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362eaee3-d79c-4cf0-a481-25d60bc67c23 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:26.228974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 553/1110] 10.222.0.26 () {60 vars in 1132 bytes} [Tue Sep 27 04:08:26 2022] POST /identity/v3/users => generated 286 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:08:28.373142 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9209bc78-618a-4bab-bac7-25ab3e37124c None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:28.373709 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9209bc78-618a-4bab-bac7-25ab3e37124c None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:28.373709 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9209bc78-618a-4bab-bac7-25ab3e37124c None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:28.375031 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 558/1111] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:08:28.381041 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ed91e9-94ba-4638-ad5d-cead8644beef None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:28.381437 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ed91e9-94ba-4638-ad5d-cead8644beef None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:28.381579 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ed91e9-94ba-4638-ad5d-cead8644beef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:28.399822 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-00ed91e9-94ba-4638-ad5d-cead8644beef None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:28.400988 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-00ed91e9-94ba-4638-ad5d-cead8644beef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:28.437197 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:28.470284 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 554/1112] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:28 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:08:28.739085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9209bc78-618a-4bab-bac7-25ab3e37124c None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:28.835955 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9209bc78-618a-4bab-bac7-25ab3e37124c None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:28.854658 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:28.868863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f70c7f7-f260-44f3-9911-1957f88e2186 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:28.869477 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f70c7f7-f260-44f3-9911-1957f88e2186 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:28.869782 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f70c7f7-f260-44f3-9911-1957f88e2186 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:28.870054 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f70c7f7-f260-44f3-9911-1957f88e2186 None admin] PATH_INFO: `/v3/roles/2d9f391485604d0b952b7c844c3a3266` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:28.871412 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f70c7f7-f260-44f3-9911-1957f88e2186 None admin] RBAC: Authorizing `identity:get_role(role_id=2d9f391485604d0b952b7c844c3a3266)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:28.872760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f70c7f7-f260-44f3-9911-1957f88e2186 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:28.874089 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 559/1113] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:08:28 2022] GET /identity/v3/roles/2d9f391485604d0b952b7c844c3a3266 => generated 223 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:28.879705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ed91e9-94ba-4638-ad5d-cead8644beef None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:28.972842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ed91e9-94ba-4638-ad5d-cead8644beef None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:29.007617 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09176e6a-ceaf-49bc-bd39-a95d1235f167 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:29.008166 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09176e6a-ceaf-49bc-bd39-a95d1235f167 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:29.008440 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09176e6a-ceaf-49bc-bd39-a95d1235f167 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:29.008709 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09176e6a-ceaf-49bc-bd39-a95d1235f167 None admin] PATH_INFO: `/v3/users/80719a0fc6a94d448db38fc6a1706e41` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:29.021645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09176e6a-ceaf-49bc-bd39-a95d1235f167 None admin] RBAC: Authorizing `identity:get_user(user_id=80719a0fc6a94d448db38fc6a1706e41)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:29.023079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09176e6a-ceaf-49bc-bd39-a95d1235f167 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:29.024610 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 555/1114] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:08:28 2022] GET /identity/v3/users/80719a0fc6a94d448db38fc6a1706e41 => generated 286 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:29.030433 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f70c7f7-f260-44f3-9911-1957f88e2186 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:29.131692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f70c7f7-f260-44f3-9911-1957f88e2186 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:29.163322 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b2f347-ce22-4ae4-8b9b-29067492280a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:29.163915 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b2f347-ce22-4ae4-8b9b-29067492280a None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:29.164195 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b2f347-ce22-4ae4-8b9b-29067492280a None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:29.164469 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b2f347-ce22-4ae4-8b9b-29067492280a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:29.216420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:29.229262 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b2f347-ce22-4ae4-8b9b-29067492280a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:29.230740 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b2f347-ce22-4ae4-8b9b-29067492280a None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:29.323105 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 560/1115] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:08:29 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 04:08:29.329044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09176e6a-ceaf-49bc-bd39-a95d1235f167 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:29.356657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:29.425880 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09176e6a-ceaf-49bc-bd39-a95d1235f167 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:29.459324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64af28d6-55bd-4561-a10d-20f00279e761 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:29.459937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64af28d6-55bd-4561-a10d-20f00279e761 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:29.460212 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64af28d6-55bd-4561-a10d-20f00279e761 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:29.460481 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64af28d6-55bd-4561-a10d-20f00279e761 None admin] PATH_INFO: `/v3/projects/e84e913545a84f13889185e1422a07a2` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:29.461566 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64af28d6-55bd-4561-a10d-20f00279e761 None admin] RBAC: Authorizing `identity:get_project(project_id=e84e913545a84f13889185e1422a07a2)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:29.463152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64af28d6-55bd-4561-a10d-20f00279e761 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:29.464938 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 556/1116] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:08:29 2022] GET /identity/v3/projects/e84e913545a84f13889185e1422a07a2 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:29.471212 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b2f347-ce22-4ae4-8b9b-29067492280a None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:29.553770 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:29.565511 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b2f347-ce22-4ae4-8b9b-29067492280a None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:29.598211 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e83baf8-5dfc-4612-bf0f-f6d67f182730 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:29.598742 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e83baf8-5dfc-4612-bf0f-f6d67f182730 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:29.599021 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e83baf8-5dfc-4612-bf0f-f6d67f182730 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:29.599290 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e83baf8-5dfc-4612-bf0f-f6d67f182730 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:29.600002 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e83baf8-5dfc-4612-bf0f-f6d67f182730 None admin] RBAC: Adding query filter params (role.id=2d9f391485604d0b952b7c844c3a3266, scope.project.id=e84e913545a84f13889185e1422a07a2, user.id=80719a0fc6a94d448db38fc6a1706e41) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:29.600307 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e83baf8-5dfc-4612-bf0f-f6d67f182730 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:29.601712 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e83baf8-5dfc-4612-bf0f-f6d67f182730 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:29.607016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 561/1117] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:08:29 2022] GET /identity/v3/role_assignments?user.id=80719a0fc6a94d448db38fc6a1706e41&scope.project.id=e84e913545a84f13889185e1422a07a2&role.id=2d9f391485604d0b952b7c844c3a3266 => generated 259 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:31.722519 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32704895-1832-4718-a75c-62d7369bd69f None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:31.723033 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32704895-1832-4718-a75c-62d7369bd69f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:31.723033 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32704895-1832-4718-a75c-62d7369bd69f None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:31.724350 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 557/1118] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:08:31.729508 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc02517-f5f2-44ba-a68c-c026ec322cfb None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:31.729944 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc02517-f5f2-44ba-a68c-c026ec322cfb None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:31.730090 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc02517-f5f2-44ba-a68c-c026ec322cfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:31.748222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-5dc02517-f5f2-44ba-a68c-c026ec322cfb None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:31.749399 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-5dc02517-f5f2-44ba-a68c-c026ec322cfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:31.813244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 562/1119] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:31 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:08:32.083878 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32704895-1832-4718-a75c-62d7369bd69f None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:32.174466 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:32.181822 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32704895-1832-4718-a75c-62d7369bd69f None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:32.213970 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535b7299-60ba-4dc7-adb9-53faab3f03d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:32.214576 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535b7299-60ba-4dc7-adb9-53faab3f03d5 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:32.214850 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535b7299-60ba-4dc7-adb9-53faab3f03d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:32.215117 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535b7299-60ba-4dc7-adb9-53faab3f03d5 None admin] PATH_INFO: `/v3/roles/2d9f391485604d0b952b7c844c3a3266` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:32.216469 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535b7299-60ba-4dc7-adb9-53faab3f03d5 None admin] RBAC: Authorizing `identity:get_role(role_id=2d9f391485604d0b952b7c844c3a3266)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:32.217830 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535b7299-60ba-4dc7-adb9-53faab3f03d5 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:32.219119 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 558/1120] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:08:32 2022] GET /identity/v3/roles/2d9f391485604d0b952b7c844c3a3266 => generated 223 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:08:32.225448 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc02517-f5f2-44ba-a68c-c026ec322cfb None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:32.302970 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:32.320636 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc02517-f5f2-44ba-a68c-c026ec322cfb None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:32.353575 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0146416-2124-4f69-896f-830c2b3ac3ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:32.354155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0146416-2124-4f69-896f-830c2b3ac3ee None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:32.354435 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0146416-2124-4f69-896f-830c2b3ac3ee None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:32.354708 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0146416-2124-4f69-896f-830c2b3ac3ee None admin] PATH_INFO: `/v3/users/80719a0fc6a94d448db38fc6a1706e41` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:32.355807 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0146416-2124-4f69-896f-830c2b3ac3ee None admin] RBAC: Authorizing `identity:get_user(user_id=80719a0fc6a94d448db38fc6a1706e41)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:32.357218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0146416-2124-4f69-896f-830c2b3ac3ee None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:32.358780 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 563/1121] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:08:32 2022] GET /identity/v3/users/80719a0fc6a94d448db38fc6a1706e41 => generated 286 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:32.364317 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535b7299-60ba-4dc7-adb9-53faab3f03d5 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:32.458963 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535b7299-60ba-4dc7-adb9-53faab3f03d5 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:32.490735 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322bf85a-6c62-49f8-8e2d-7b1740a913d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:32.491340 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322bf85a-6c62-49f8-8e2d-7b1740a913d7 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:32.491612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322bf85a-6c62-49f8-8e2d-7b1740a913d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:32.491881 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322bf85a-6c62-49f8-8e2d-7b1740a913d7 None admin] PATH_INFO: `/v3/projects/e84e913545a84f13889185e1422a07a2` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:32.492951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322bf85a-6c62-49f8-8e2d-7b1740a913d7 None admin] RBAC: Authorizing `identity:get_project(project_id=e84e913545a84f13889185e1422a07a2)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:32.494391 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322bf85a-6c62-49f8-8e2d-7b1740a913d7 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:32.495975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 559/1122] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:08:32 2022] GET /identity/v3/projects/e84e913545a84f13889185e1422a07a2 => generated 311 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:32.501857 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0146416-2124-4f69-896f-830c2b3ac3ee None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:32.596381 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0146416-2124-4f69-896f-830c2b3ac3ee None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:32.628527 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa35d32-7e32-4b93-9e06-a2d77833c861 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:32.629098 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa35d32-7e32-4b93-9e06-a2d77833c861 None admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:32.629374 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa35d32-7e32-4b93-9e06-a2d77833c861 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:32.629641 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa35d32-7e32-4b93-9e06-a2d77833c861 None admin] PATH_INFO: `/v3/projects/e84e913545a84f13889185e1422a07a2/users/80719a0fc6a94d448db38fc6a1706e41/roles/2d9f391485604d0b952b7c844c3a3266` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:32.632535 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa35d32-7e32-4b93-9e06-a2d77833c861 None admin] RBAC: Authorizing `identity:create_grant(project_id=e84e913545a84f13889185e1422a07a2, user_id=80719a0fc6a94d448db38fc6a1706e41, role_id=2d9f391485604d0b952b7c844c3a3266)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:32.633955 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa35d32-7e32-4b93-9e06-a2d77833c861 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:32.646613 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 564/1123] 10.222.0.26 () {58 vars in 1439 bytes} [Tue Sep 27 04:08:32 2022] PUT /identity/v3/projects/e84e913545a84f13889185e1422a07a2/users/80719a0fc6a94d448db38fc6a1706e41/roles/2d9f391485604d0b952b7c844c3a3266 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:08:34.831011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0458c49-0125-4fce-af85-935f035537ff None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:34.831546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0458c49-0125-4fce-af85-935f035537ff None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:34.831546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0458c49-0125-4fce-af85-935f035537ff None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:34.832865 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 560/1124] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:08:34.837971 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999626a5-f955-425e-b06c-06a24a243d76 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:34.838345 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999626a5-f955-425e-b06c-06a24a243d76 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:34.838520 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999626a5-f955-425e-b06c-06a24a243d76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:34.856541 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-999626a5-f955-425e-b06c-06a24a243d76 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:34.857710 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-999626a5-f955-425e-b06c-06a24a243d76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:34.860397 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:34.922246 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 565/1125] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:34 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:08:35.195086 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0458c49-0125-4fce-af85-935f035537ff None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:35.225483 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:35.295878 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0458c49-0125-4fce-af85-935f035537ff None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:35.327128 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1e38f8-1cc6-4f4c-888c-06b4227796b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:35.327738 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1e38f8-1cc6-4f4c-888c-06b4227796b2 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:35.328008 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1e38f8-1cc6-4f4c-888c-06b4227796b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:35.328272 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1e38f8-1cc6-4f4c-888c-06b4227796b2 None admin] PATH_INFO: `/v3/roles/2d9f391485604d0b952b7c844c3a3266` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:35.329601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1e38f8-1cc6-4f4c-888c-06b4227796b2 None admin] RBAC: Authorizing `identity:get_role(role_id=2d9f391485604d0b952b7c844c3a3266)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:35.330963 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1e38f8-1cc6-4f4c-888c-06b4227796b2 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:35.332256 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 561/1126] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:08:35 2022] GET /identity/v3/roles/2d9f391485604d0b952b7c844c3a3266 => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:35.338703 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999626a5-f955-425e-b06c-06a24a243d76 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:35.434961 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999626a5-f955-425e-b06c-06a24a243d76 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:35.466860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28251c9-6b73-4049-8dda-73d2305f8036 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:35.467434 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28251c9-6b73-4049-8dda-73d2305f8036 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:35.467710 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28251c9-6b73-4049-8dda-73d2305f8036 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:35.467978 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28251c9-6b73-4049-8dda-73d2305f8036 None admin] PATH_INFO: `/v3/users/80719a0fc6a94d448db38fc6a1706e41` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:35.469063 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28251c9-6b73-4049-8dda-73d2305f8036 None admin] RBAC: Authorizing `identity:get_user(user_id=80719a0fc6a94d448db38fc6a1706e41)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:35.470490 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28251c9-6b73-4049-8dda-73d2305f8036 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:35.472024 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 566/1127] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:08:35 2022] GET /identity/v3/users/80719a0fc6a94d448db38fc6a1706e41 => generated 286 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:35.478245 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1e38f8-1cc6-4f4c-888c-06b4227796b2 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:35.572118 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:35.575327 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1e38f8-1cc6-4f4c-888c-06b4227796b2 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:35.609106 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3367c57e-3fa5-4bad-a12f-29828558999c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:35.609693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3367c57e-3fa5-4bad-a12f-29828558999c None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:35.609992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3367c57e-3fa5-4bad-a12f-29828558999c None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:35.610255 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3367c57e-3fa5-4bad-a12f-29828558999c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:35.673334 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3367c57e-3fa5-4bad-a12f-29828558999c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:35.674784 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3367c57e-3fa5-4bad-a12f-29828558999c None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:35.766735 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:35.771193 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 562/1128] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:08:35 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 04:08:35.776860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28251c9-6b73-4049-8dda-73d2305f8036 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:35.787480 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:35.878762 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28251c9-6b73-4049-8dda-73d2305f8036 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:35.914843 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596372f1-005c-473f-8c71-42368e76587b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:35.915458 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596372f1-005c-473f-8c71-42368e76587b None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:35.915741 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596372f1-005c-473f-8c71-42368e76587b None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:35.916016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596372f1-005c-473f-8c71-42368e76587b None admin] PATH_INFO: `/v3/projects/e84e913545a84f13889185e1422a07a2` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:35.917121 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596372f1-005c-473f-8c71-42368e76587b None admin] RBAC: Authorizing `identity:get_project(project_id=e84e913545a84f13889185e1422a07a2)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:35.918560 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596372f1-005c-473f-8c71-42368e76587b None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:35.920172 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 567/1129] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:08:35 2022] GET /identity/v3/projects/e84e913545a84f13889185e1422a07a2 => generated 311 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:35.926374 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3367c57e-3fa5-4bad-a12f-29828558999c None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:36.022218 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3367c57e-3fa5-4bad-a12f-29828558999c None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:36.054334 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298dcff3-dd69-49e7-86c4-62ab045b0208 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:36.054915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298dcff3-dd69-49e7-86c4-62ab045b0208 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:36.055186 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298dcff3-dd69-49e7-86c4-62ab045b0208 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:36.055452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298dcff3-dd69-49e7-86c4-62ab045b0208 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:36.056156 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298dcff3-dd69-49e7-86c4-62ab045b0208 None admin] RBAC: Adding query filter params (role.id=2d9f391485604d0b952b7c844c3a3266, scope.project.id=e84e913545a84f13889185e1422a07a2, user.id=80719a0fc6a94d448db38fc6a1706e41) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:36.056456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298dcff3-dd69-49e7-86c4-62ab045b0208 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:36.057876 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298dcff3-dd69-49e7-86c4-62ab045b0208 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:36.063275 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 563/1130] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:08:35 2022] GET /identity/v3/role_assignments?user.id=80719a0fc6a94d448db38fc6a1706e41&scope.project.id=e84e913545a84f13889185e1422a07a2&role.id=2d9f391485604d0b952b7c844c3a3266 => generated 608 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:38.186858 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817df3d6-49b7-4b97-be10-9fff26c9c4b6 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:38.187390 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817df3d6-49b7-4b97-be10-9fff26c9c4b6 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:38.187390 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817df3d6-49b7-4b97-be10-9fff26c9c4b6 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:38.188697 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 568/1131] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:08:38.194620 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3384c9bb-8724-4095-ad12-f92c71d72612 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:38.195025 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3384c9bb-8724-4095-ad12-f92c71d72612 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:38.195164 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3384c9bb-8724-4095-ad12-f92c71d72612 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:38.213182 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-3384c9bb-8724-4095-ad12-f92c71d72612 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:38.214366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-3384c9bb-8724-4095-ad12-f92c71d72612 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:38.270769 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:38.281268 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 564/1132] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:08:38.552515 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817df3d6-49b7-4b97-be10-9fff26c9c4b6 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:38.627211 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:38.650619 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817df3d6-49b7-4b97-be10-9fff26c9c4b6 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:38.683430 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab805ae-f0ee-4fb1-98f6-648112b8d999 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:38.684040 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab805ae-f0ee-4fb1-98f6-648112b8d999 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:38.684322 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab805ae-f0ee-4fb1-98f6-648112b8d999 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:38.684597 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab805ae-f0ee-4fb1-98f6-648112b8d999 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:38.685690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab805ae-f0ee-4fb1-98f6-648112b8d999 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:38.687137 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab805ae-f0ee-4fb1-98f6-648112b8d999 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:38.688654 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 569/1133] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:08:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:38.694368 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3384c9bb-8724-4095-ad12-f92c71d72612 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:38.787618 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3384c9bb-8724-4095-ad12-f92c71d72612 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:38.831888 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009895dc-8739-4c5f-a4a0-bc23dc8d6650 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:38.832413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009895dc-8739-4c5f-a4a0-bc23dc8d6650 None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:38.832692 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009895dc-8739-4c5f-a4a0-bc23dc8d6650 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:38.832964 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009895dc-8739-4c5f-a4a0-bc23dc8d6650 None admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:38.833602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009895dc-8739-4c5f-a4a0-bc23dc8d6650 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:38.835057 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009895dc-8739-4c5f-a4a0-bc23dc8d6650 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:38.860609 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 565/1134] 10.222.0.26 () {60 vars in 1141 bytes} [Tue Sep 27 04:08:38 2022] POST /identity/v3/projects => generated 311 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:08:40.993213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c6792b-71a5-4834-89fc-462a4e87c794 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:40.993801 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c6792b-71a5-4834-89fc-462a4e87c794 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:40.993801 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c6792b-71a5-4834-89fc-462a4e87c794 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:40.995081 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 570/1135] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:08:41.000239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6741945-3f72-45d0-9e34-668e8e14caca None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:41.000659 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6741945-3f72-45d0-9e34-668e8e14caca None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:41.000791 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6741945-3f72-45d0-9e34-668e8e14caca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:41.018942 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-b6741945-3f72-45d0-9e34-668e8e14caca None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:41.020108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-b6741945-3f72-45d0-9e34-668e8e14caca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:41.060590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:41.084745 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 566/1136] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:40 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:08:41.355244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c6792b-71a5-4834-89fc-462a4e87c794 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:41.452566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c6792b-71a5-4834-89fc-462a4e87c794 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:41.485509 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06225ff-b9ea-405e-bbbf-4420ceeff6ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:41.486156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06225ff-b9ea-405e-bbbf-4420ceeff6ae None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:41.486437 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06225ff-b9ea-405e-bbbf-4420ceeff6ae None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:41.486709 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06225ff-b9ea-405e-bbbf-4420ceeff6ae None admin] PATH_INFO: `/v3/domains/default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:41.487813 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06225ff-b9ea-405e-bbbf-4420ceeff6ae None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:41.489239 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06225ff-b9ea-405e-bbbf-4420ceeff6ae None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:41.490810 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 571/1137] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:08:41 2022] GET /identity/v3/domains/default => generated 199 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:41.496552 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6741945-3f72-45d0-9e34-668e8e14caca None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:41.590633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6741945-3f72-45d0-9e34-668e8e14caca None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:41.622134 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f584e51c-81c2-4073-931e-3f8576c8e81c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:41.622663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f584e51c-81c2-4073-931e-3f8576c8e81c None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:41.622940 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f584e51c-81c2-4073-931e-3f8576c8e81c None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:41.623209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f584e51c-81c2-4073-931e-3f8576c8e81c None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:41.623854 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f584e51c-81c2-4073-931e-3f8576c8e81c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:41.625257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f584e51c-81c2-4073-931e-3f8576c8e81c None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:41.646397 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 567/1138] 10.222.0.26 () {60 vars in 1132 bytes} [Tue Sep 27 04:08:41 2022] POST /identity/v3/users => generated 286 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:08:43.797578 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fd0ea2-dfe2-493e-b253-bcaecb37e5d7 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:43.798117 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fd0ea2-dfe2-493e-b253-bcaecb37e5d7 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:43.798208 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fd0ea2-dfe2-493e-b253-bcaecb37e5d7 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:43.799498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 572/1139] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:08:43.805332 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2f4db1-020b-4f63-9421-512aca857404 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:43.805774 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2f4db1-020b-4f63-9421-512aca857404 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:43.805919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2f4db1-020b-4f63-9421-512aca857404 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:43.824885 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-fd2f4db1-020b-4f63-9421-512aca857404 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:43.826088 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-fd2f4db1-020b-4f63-9421-512aca857404 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:43.851040 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:43.891327 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 568/1140] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:08:44.165047 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fd0ea2-dfe2-493e-b253-bcaecb37e5d7 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:44.179488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:44.264374 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fd0ea2-dfe2-493e-b253-bcaecb37e5d7 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:44.296929 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf78f3f-1846-4e1c-8c74-e3bc421b12c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:44.297543 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf78f3f-1846-4e1c-8c74-e3bc421b12c4 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:44.297849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf78f3f-1846-4e1c-8c74-e3bc421b12c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:44.298122 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf78f3f-1846-4e1c-8c74-e3bc421b12c4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:44.308352 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf78f3f-1846-4e1c-8c74-e3bc421b12c4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:08:44.313159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-7cf78f3f-1846-4e1c-8c74-e3bc421b12c4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:08:44.313159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:44.313159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:44.313159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:44.313159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:44.313159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:44.313159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:44.313159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:44.313159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:44.313159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:44.313159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:44.313159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:44.313159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:08:44.313159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:08:44.313159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:08:44.313159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:08:44.314357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:08:44.314357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:08:44.314357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:08:44.314357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:08:44.314357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:44.314357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:44.314357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:44.314357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:44.314357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:44.314357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:44.314357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:44.314357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:44.314357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:44.314357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:44.314357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:44.314357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:44.315643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:44.315643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:44.315643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:44.315643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:44.315643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:08:44.315643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:08:44.315643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:08:44.315643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:08:44.315643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:08:44.315643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:08:44.315643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:08:44.315643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:44.315643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 573/1141] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:08:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:08:44.320886 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2f4db1-020b-4f63-9421-512aca857404 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:44.414727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2f4db1-020b-4f63-9421-512aca857404 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:44.448093 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c3c32e-db8f-417a-a640-b9e66dac816f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:44.448622 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c3c32e-db8f-417a-a640-b9e66dac816f None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:44.448895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c3c32e-db8f-417a-a640-b9e66dac816f None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:44.449164 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c3c32e-db8f-417a-a640-b9e66dac816f None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:44.449841 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c3c32e-db8f-417a-a640-b9e66dac816f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:44.450140 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c3c32e-db8f-417a-a640-b9e66dac816f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:44.451482 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c3c32e-db8f-417a-a640-b9e66dac816f None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:44.459125 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 569/1142] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:08:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:44.465004 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf78f3f-1846-4e1c-8c74-e3bc421b12c4 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:44.533232 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:44.561615 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf78f3f-1846-4e1c-8c74-e3bc421b12c4 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:44.594273 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df8099e-5be9-47f5-901f-10ebfe61d18d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:44.594878 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df8099e-5be9-47f5-901f-10ebfe61d18d None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:44.595159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df8099e-5be9-47f5-901f-10ebfe61d18d None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:44.595433 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df8099e-5be9-47f5-901f-10ebfe61d18d None admin] PATH_INFO: `/v3/users/4771588d97ef4c749bbaaa3e2dfc5d4e` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:44.608345 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df8099e-5be9-47f5-901f-10ebfe61d18d None admin] RBAC: Authorizing `identity:get_user(user_id=4771588d97ef4c749bbaaa3e2dfc5d4e)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:44.609778 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df8099e-5be9-47f5-901f-10ebfe61d18d None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:44.611341 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 574/1143] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:08:44 2022] GET /identity/v3/users/4771588d97ef4c749bbaaa3e2dfc5d4e => generated 286 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:44.617603 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c3c32e-db8f-417a-a640-b9e66dac816f None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:44.642212 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:44.712654 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c3c32e-db8f-417a-a640-b9e66dac816f None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:44.744546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51784245-8cee-4dd7-84b6-768760555917 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:44.745131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51784245-8cee-4dd7-84b6-768760555917 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:44.745404 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51784245-8cee-4dd7-84b6-768760555917 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:44.745671 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51784245-8cee-4dd7-84b6-768760555917 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:44.808702 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51784245-8cee-4dd7-84b6-768760555917 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:44.810142 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51784245-8cee-4dd7-84b6-768760555917 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:44.831938 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:44.902513 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 570/1144] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:08:44 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 285 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 04:08:44.908309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df8099e-5be9-47f5-901f-10ebfe61d18d None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:45.004308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df8099e-5be9-47f5-901f-10ebfe61d18d None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:45.031822 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:45.038071 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4066a9f-5e9a-43b1-a6bb-f9804dfb25d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:45.038678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4066a9f-5e9a-43b1-a6bb-f9804dfb25d6 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:45.038958 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4066a9f-5e9a-43b1-a6bb-f9804dfb25d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:45.039230 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4066a9f-5e9a-43b1-a6bb-f9804dfb25d6 None admin] PATH_INFO: `/v3/projects/a2db22afecb84cbcb68617116ca6582d` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:45.040315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4066a9f-5e9a-43b1-a6bb-f9804dfb25d6 None admin] RBAC: Authorizing `identity:get_project(project_id=a2db22afecb84cbcb68617116ca6582d)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:45.041711 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4066a9f-5e9a-43b1-a6bb-f9804dfb25d6 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:45.043311 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 575/1145] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:08:44 2022] GET /identity/v3/projects/a2db22afecb84cbcb68617116ca6582d => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:45.049219 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51784245-8cee-4dd7-84b6-768760555917 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:45.143152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51784245-8cee-4dd7-84b6-768760555917 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:45.165844 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:45.176624 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2fb8c6-1aeb-43e9-a0e9-9e3b39c80ee7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:45.177247 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2fb8c6-1aeb-43e9-a0e9-9e3b39c80ee7 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:45.177519 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2fb8c6-1aeb-43e9-a0e9-9e3b39c80ee7 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:45.177811 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2fb8c6-1aeb-43e9-a0e9-9e3b39c80ee7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:45.178521 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2fb8c6-1aeb-43e9-a0e9-9e3b39c80ee7 None admin] RBAC: Adding query filter params (role.id=e6a31562a18649f0809202c36a8e18ef, scope.project.id=a2db22afecb84cbcb68617116ca6582d, user.id=4771588d97ef4c749bbaaa3e2dfc5d4e) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:45.178824 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2fb8c6-1aeb-43e9-a0e9-9e3b39c80ee7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:45.180228 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2fb8c6-1aeb-43e9-a0e9-9e3b39c80ee7 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:45.185308 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 571/1146] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:08:45 2022] GET /identity/v3/role_assignments?user.id=4771588d97ef4c749bbaaa3e2dfc5d4e&scope.project.id=a2db22afecb84cbcb68617116ca6582d&role.id=e6a31562a18649f0809202c36a8e18ef => generated 259 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:47.259229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7518510f-a1ec-45e3-aa80-4fad32b3cc77 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:47.259764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7518510f-a1ec-45e3-aa80-4fad32b3cc77 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:47.259764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7518510f-a1ec-45e3-aa80-4fad32b3cc77 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:47.261073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 576/1147] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:08:47.266183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a75e88-b397-4992-8872-864f9c20cbd6 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:47.266603 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a75e88-b397-4992-8872-864f9c20cbd6 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:47.266718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a75e88-b397-4992-8872-864f9c20cbd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:47.284116 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-13a75e88-b397-4992-8872-864f9c20cbd6 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:47.285270 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-13a75e88-b397-4992-8872-864f9c20cbd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:47.348386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 572/1148] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:08:47.615901 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7518510f-a1ec-45e3-aa80-4fad32b3cc77 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:47.712589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7518510f-a1ec-45e3-aa80-4fad32b3cc77 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:47.745111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b34a492-4856-4938-93c9-55a7bcb61207 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:47.745717 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b34a492-4856-4938-93c9-55a7bcb61207 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:47.746024 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b34a492-4856-4938-93c9-55a7bcb61207 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:47.746297 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b34a492-4856-4938-93c9-55a7bcb61207 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:47.755914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b34a492-4856-4938-93c9-55a7bcb61207 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:08:47.760962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-9b34a492-4856-4938-93c9-55a7bcb61207 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:08:47.760962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:47.760962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:47.760962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:47.760962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:47.760962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:47.760962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:47.760962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:47.760962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:47.760962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:47.760962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:47.760962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:47.760962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:08:47.760962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:08:47.760962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:08:47.760962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:08:47.762074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:08:47.762074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:08:47.762074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:08:47.762074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:08:47.762074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:47.762074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:47.762074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:47.762074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:47.762074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:47.762074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:47.762074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:47.762074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:47.762074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:47.762074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:47.762074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:47.762074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:47.762925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:47.762925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:47.762925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:47.762925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:47.762925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:08:47.762925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:08:47.762925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:08:47.762925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:08:47.762925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:08:47.762925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:08:47.762925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:08:47.762925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:47.762925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 577/1149] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:08:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:08:47.767818 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a75e88-b397-4992-8872-864f9c20cbd6 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:47.861427 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a75e88-b397-4992-8872-864f9c20cbd6 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:47.878133 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:47.894814 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724b33c9-4053-435c-a53a-692f4a247893 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:47.895412 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724b33c9-4053-435c-a53a-692f4a247893 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:47.895701 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724b33c9-4053-435c-a53a-692f4a247893 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:47.895975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724b33c9-4053-435c-a53a-692f4a247893 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:47.896630 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-724b33c9-4053-435c-a53a-692f4a247893 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:47.896929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-724b33c9-4053-435c-a53a-692f4a247893 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:47.898321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-724b33c9-4053-435c-a53a-692f4a247893 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:47.906245 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 573/1150] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:08:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:47.912813 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b34a492-4856-4938-93c9-55a7bcb61207 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:47.971027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:48.011194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b34a492-4856-4938-93c9-55a7bcb61207 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:48.043635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57cbff6-d2cb-4438-b2af-066f1395f581 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:48.044242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57cbff6-d2cb-4438-b2af-066f1395f581 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:48.044520 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57cbff6-d2cb-4438-b2af-066f1395f581 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:48.044789 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57cbff6-d2cb-4438-b2af-066f1395f581 None admin] PATH_INFO: `/v3/users/4771588d97ef4c749bbaaa3e2dfc5d4e` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:48.045912 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57cbff6-d2cb-4438-b2af-066f1395f581 None admin] RBAC: Authorizing `identity:get_user(user_id=4771588d97ef4c749bbaaa3e2dfc5d4e)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:48.047319 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57cbff6-d2cb-4438-b2af-066f1395f581 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:48.048848 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 578/1151] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:08:47 2022] GET /identity/v3/users/4771588d97ef4c749bbaaa3e2dfc5d4e => generated 286 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:48.054741 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724b33c9-4053-435c-a53a-692f4a247893 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:48.150209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724b33c9-4053-435c-a53a-692f4a247893 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:48.182281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7663f4fe-7cc0-4281-a470-4aa06f8e8488 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:48.182890 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7663f4fe-7cc0-4281-a470-4aa06f8e8488 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:48.183168 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7663f4fe-7cc0-4281-a470-4aa06f8e8488 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:48.183439 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7663f4fe-7cc0-4281-a470-4aa06f8e8488 None admin] PATH_INFO: `/v3/projects/a2db22afecb84cbcb68617116ca6582d` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:48.184524 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7663f4fe-7cc0-4281-a470-4aa06f8e8488 None admin] RBAC: Authorizing `identity:get_project(project_id=a2db22afecb84cbcb68617116ca6582d)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:48.185955 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7663f4fe-7cc0-4281-a470-4aa06f8e8488 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:48.187532 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 574/1152] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:08:48 2022] GET /identity/v3/projects/a2db22afecb84cbcb68617116ca6582d => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:48.193310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57cbff6-d2cb-4438-b2af-066f1395f581 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:48.289800 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57cbff6-d2cb-4438-b2af-066f1395f581 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:48.314337 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:48.323529 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f53a406-536a-41ef-91dc-03e3a81e2cf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:48.324169 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f53a406-536a-41ef-91dc-03e3a81e2cf7 None admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:48.324446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f53a406-536a-41ef-91dc-03e3a81e2cf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:48.324714 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f53a406-536a-41ef-91dc-03e3a81e2cf7 None admin] PATH_INFO: `/v3/projects/a2db22afecb84cbcb68617116ca6582d/users/4771588d97ef4c749bbaaa3e2dfc5d4e/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:48.326592 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f53a406-536a-41ef-91dc-03e3a81e2cf7 None admin] RBAC: Authorizing `identity:create_grant(project_id=a2db22afecb84cbcb68617116ca6582d, user_id=4771588d97ef4c749bbaaa3e2dfc5d4e, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:48.327994 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f53a406-536a-41ef-91dc-03e3a81e2cf7 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:48.340244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 579/1153] 10.222.0.26 () {58 vars in 1439 bytes} [Tue Sep 27 04:08:48 2022] PUT /identity/v3/projects/a2db22afecb84cbcb68617116ca6582d/users/4771588d97ef4c749bbaaa3e2dfc5d4e/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:08:50.453385 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54e5dd6-78e4-48f7-a8e8-30b0c5426727 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:50.453936 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54e5dd6-78e4-48f7-a8e8-30b0c5426727 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:50.453936 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54e5dd6-78e4-48f7-a8e8-30b0c5426727 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:50.455272 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 575/1154] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:08:50.460915 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec91096-1c1a-4271-bd7f-95cf8ff78860 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:50.461305 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec91096-1c1a-4271-bd7f-95cf8ff78860 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:50.461448 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec91096-1c1a-4271-bd7f-95cf8ff78860 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:50.479565 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-fec91096-1c1a-4271-bd7f-95cf8ff78860 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:50.480748 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-fec91096-1c1a-4271-bd7f-95cf8ff78860 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:50.543378 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 580/1155] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:50 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:08:50.812036 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54e5dd6-78e4-48f7-a8e8-30b0c5426727 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:50.860281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:50.909908 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54e5dd6-78e4-48f7-a8e8-30b0c5426727 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:50.940992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b6a283-dc1b-4097-a63c-49beff07343a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:50.941598 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b6a283-dc1b-4097-a63c-49beff07343a None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:50.941901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b6a283-dc1b-4097-a63c-49beff07343a None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:50.942172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b6a283-dc1b-4097-a63c-49beff07343a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:50.952830 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b6a283-dc1b-4097-a63c-49beff07343a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:08:50.959680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-13b6a283-dc1b-4097-a63c-49beff07343a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:08:50.959680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:50.959680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:50.959680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:50.959680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:50.959680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:50.959680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:50.959680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:50.959680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:50.959680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:50.959680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:50.959680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:50.959680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:08:50.959680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:08:50.959680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:08:50.959680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:08:50.960837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:08:50.960837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:08:50.960837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:08:50.960837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:08:50.960837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:50.960837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:50.960837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:50.960837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:50.960837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:50.960837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:50.960837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:50.960837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:50.960837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:50.960837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:50.960837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:50.960837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:50.962195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:50.962195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:50.962195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:50.962195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:50.962195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:08:50.962195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:08:50.962195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:08:50.962195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:08:50.962195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:08:50.962195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:08:50.962195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:08:50.962195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:08:50.962195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 576/1156] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:08:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:08:50.968221 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec91096-1c1a-4271-bd7f-95cf8ff78860 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:51.062682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec91096-1c1a-4271-bd7f-95cf8ff78860 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:51.079458 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:51.096932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0625c849-9492-43b0-a459-966dea91e51d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:51.097521 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0625c849-9492-43b0-a459-966dea91e51d None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:51.097824 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0625c849-9492-43b0-a459-966dea91e51d None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:51.098097 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0625c849-9492-43b0-a459-966dea91e51d None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:51.098753 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0625c849-9492-43b0-a459-966dea91e51d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:51.099051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0625c849-9492-43b0-a459-966dea91e51d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:51.100393 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0625c849-9492-43b0-a459-966dea91e51d None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:51.108066 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 581/1157] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:08:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:51.114037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b6a283-dc1b-4097-a63c-49beff07343a None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:51.210382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b6a283-dc1b-4097-a63c-49beff07343a None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:51.218707 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:51.243107 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c03da7d-d6ed-4ebf-9ddc-bf017f477320 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:51.243718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c03da7d-d6ed-4ebf-9ddc-bf017f477320 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:51.243991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c03da7d-d6ed-4ebf-9ddc-bf017f477320 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:51.244256 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c03da7d-d6ed-4ebf-9ddc-bf017f477320 None admin] PATH_INFO: `/v3/users/4771588d97ef4c749bbaaa3e2dfc5d4e` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:51.245328 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c03da7d-d6ed-4ebf-9ddc-bf017f477320 None admin] RBAC: Authorizing `identity:get_user(user_id=4771588d97ef4c749bbaaa3e2dfc5d4e)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:51.246759 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c03da7d-d6ed-4ebf-9ddc-bf017f477320 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:51.248282 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 577/1158] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:08:51 2022] GET /identity/v3/users/4771588d97ef4c749bbaaa3e2dfc5d4e => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:51.253835 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0625c849-9492-43b0-a459-966dea91e51d None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:51.352191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0625c849-9492-43b0-a459-966dea91e51d None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:51.384254 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfca2252-59a2-496d-8db0-3872a0f96998 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:51.384836 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfca2252-59a2-496d-8db0-3872a0f96998 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:51.385109 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfca2252-59a2-496d-8db0-3872a0f96998 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:51.385375 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfca2252-59a2-496d-8db0-3872a0f96998 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:51.429325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:51.450377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfca2252-59a2-496d-8db0-3872a0f96998 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:51.451800 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfca2252-59a2-496d-8db0-3872a0f96998 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:51.544514 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 582/1159] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:08:51 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 04:08:51.550418 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c03da7d-d6ed-4ebf-9ddc-bf017f477320 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:51.651366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c03da7d-d6ed-4ebf-9ddc-bf017f477320 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:51.683398 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2e7b43-fe92-4a2c-8dfa-2cd64c6c1aef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:51.684009 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2e7b43-fe92-4a2c-8dfa-2cd64c6c1aef None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:51.684286 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2e7b43-fe92-4a2c-8dfa-2cd64c6c1aef None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:51.684558 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2e7b43-fe92-4a2c-8dfa-2cd64c6c1aef None admin] PATH_INFO: `/v3/projects/a2db22afecb84cbcb68617116ca6582d` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:51.685644 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2e7b43-fe92-4a2c-8dfa-2cd64c6c1aef None admin] RBAC: Authorizing `identity:get_project(project_id=a2db22afecb84cbcb68617116ca6582d)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:51.687089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2e7b43-fe92-4a2c-8dfa-2cd64c6c1aef None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:51.688690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 578/1160] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:08:51 2022] GET /identity/v3/projects/a2db22afecb84cbcb68617116ca6582d => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:51.694806 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfca2252-59a2-496d-8db0-3872a0f96998 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:51.768259 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:51.789737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfca2252-59a2-496d-8db0-3872a0f96998 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:51.823603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bef313-e1d8-4c84-ac08-ce86fd9661db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:51.824185 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bef313-e1d8-4c84-ac08-ce86fd9661db None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:51.824465 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bef313-e1d8-4c84-ac08-ce86fd9661db None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:51.824734 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bef313-e1d8-4c84-ac08-ce86fd9661db None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:51.825445 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bef313-e1d8-4c84-ac08-ce86fd9661db None admin] RBAC: Adding query filter params (role.id=e6a31562a18649f0809202c36a8e18ef, scope.project.id=a2db22afecb84cbcb68617116ca6582d, user.id=4771588d97ef4c749bbaaa3e2dfc5d4e) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:51.825774 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bef313-e1d8-4c84-ac08-ce86fd9661db None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:51.827181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bef313-e1d8-4c84-ac08-ce86fd9661db None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:51.832636 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 583/1161] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:08:51 2022] GET /identity/v3/role_assignments?user.id=4771588d97ef4c749bbaaa3e2dfc5d4e&scope.project.id=a2db22afecb84cbcb68617116ca6582d&role.id=e6a31562a18649f0809202c36a8e18ef => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:53.975425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f06c1c1-0b0a-419c-86ec-142be4e767fa None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:53.975981 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f06c1c1-0b0a-419c-86ec-142be4e767fa None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:53.975981 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f06c1c1-0b0a-419c-86ec-142be4e767fa None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:53.977270 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 579/1162] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:08:53.982396 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7088e6-10c8-4632-ab64-543472ecba7f None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:53.982791 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7088e6-10c8-4632-ab64-543472ecba7f None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:53.982943 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7088e6-10c8-4632-ab64-543472ecba7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:54.000873 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-fd7088e6-10c8-4632-ab64-543472ecba7f None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:54.002069 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-fd7088e6-10c8-4632-ab64-543472ecba7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:54.064575 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 584/1163] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:53 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:08:54.326917 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f06c1c1-0b0a-419c-86ec-142be4e767fa None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:54.373533 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:54.427966 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f06c1c1-0b0a-419c-86ec-142be4e767fa None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:54.464402 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eceda1d5-9803-4ff7-887b-6530aa5f5f6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:54.464996 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eceda1d5-9803-4ff7-887b-6530aa5f5f6f None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:54.465271 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eceda1d5-9803-4ff7-887b-6530aa5f5f6f None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:54.465540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eceda1d5-9803-4ff7-887b-6530aa5f5f6f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:54.529390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eceda1d5-9803-4ff7-887b-6530aa5f5f6f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:54.530846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eceda1d5-9803-4ff7-887b-6530aa5f5f6f None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:54.572657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:54.624936 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 580/1164] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:08:54 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 04:08:54.630902 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7088e6-10c8-4632-ab64-543472ecba7f None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:54.698063 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:54.727137 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7088e6-10c8-4632-ab64-543472ecba7f None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:54.759174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e13652-afd4-415a-8fb6-b8cceda086ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:54.759790 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e13652-afd4-415a-8fb6-b8cceda086ea None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:54.760071 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e13652-afd4-415a-8fb6-b8cceda086ea None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:54.760346 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e13652-afd4-415a-8fb6-b8cceda086ea None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:54.766777 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e13652-afd4-415a-8fb6-b8cceda086ea None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:54.768193 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e13652-afd4-415a-8fb6-b8cceda086ea None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:54.779840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-a1e13652-afd4-415a-8fb6-b8cceda086ea None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 27 04:08:54.779840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:54.779840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Sep 27 04:08:54.779840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Sep 27 04:08:54.779840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:08:54.779840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:08:54.779840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:08:54.779840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:08:54.779840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Sep 27 04:08:54.779840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:54.779840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:08:54.779840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:54.779840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:08:54.779840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:08:54.779840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:08:54.779840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:08:54.779840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:08:54.781036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:08:54.781036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:08:54.781036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:08:54.781036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:08:54.781036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:08:54.781036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:08:54.781036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:08:54.781036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:08:54.781036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:08:54.781036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:08:54.781036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:08:54.781036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:08:54.781036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:08:54.781036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:08:54.781036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:08:54.781036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:08:54.782154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:08:54.782154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:08:54.782154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:08:54.782154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:08:54.782154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:08:54.782154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:08:54.782154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:08:54.782154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:08:54.782154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:08:54.782154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:08:54.782154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Sep 27 04:08:54.782154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:08:54.782154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 27 04:08:54.782154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:08:54.782154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 585/1165] 10.222.0.26 () {56 vars in 1117 bytes} [Tue Sep 27 04:08:54 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:08:54.787323 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eceda1d5-9803-4ff7-887b-6530aa5f5f6f None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:54.880839 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eceda1d5-9803-4ff7-887b-6530aa5f5f6f None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:54.913868 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3d617c-ede6-4a23-ba77-4d95217b5fb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:54.914448 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3d617c-ede6-4a23-ba77-4d95217b5fb7 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:54.914725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3d617c-ede6-4a23-ba77-4d95217b5fb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:54.914997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3d617c-ede6-4a23-ba77-4d95217b5fb7 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:54.915625 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3d617c-ede6-4a23-ba77-4d95217b5fb7 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:08:54.915923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3d617c-ede6-4a23-ba77-4d95217b5fb7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:54.917264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3d617c-ede6-4a23-ba77-4d95217b5fb7 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:54.924101 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 581/1166] 10.222.0.26 () {56 vars in 1115 bytes} [Tue Sep 27 04:08:54 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:08:57.021171 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6c0f8c-cc41-4e4f-adf8-5c30c81ec6fa None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:57.021705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6c0f8c-cc41-4e4f-adf8-5c30c81ec6fa None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:57.022177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6c0f8c-cc41-4e4f-adf8-5c30c81ec6fa None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:57.023991 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 586/1167] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:08:57.029407 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339afa7f-d1c2-4dfd-b47f-19d24956afd9 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:57.029821 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339afa7f-d1c2-4dfd-b47f-19d24956afd9 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:57.030044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339afa7f-d1c2-4dfd-b47f-19d24956afd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:57.047593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-339afa7f-d1c2-4dfd-b47f-19d24956afd9 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:57.048758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-339afa7f-d1c2-4dfd-b47f-19d24956afd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:57.050694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:57.113085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 582/1168] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:08:57.387200 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6c0f8c-cc41-4e4f-adf8-5c30c81ec6fa None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:08:57.485674 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6c0f8c-cc41-4e4f-adf8-5c30c81ec6fa None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:08:57.513119 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:08:57.518282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e196cf1-67fe-4c0e-b2bf-9b83296e9738 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:08:57.518869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e196cf1-67fe-4c0e-b2bf-9b83296e9738 None admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:57.519143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e196cf1-67fe-4c0e-b2bf-9b83296e9738 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:57.519408 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e196cf1-67fe-4c0e-b2bf-9b83296e9738 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:57.520056 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e196cf1-67fe-4c0e-b2bf-9b83296e9738 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:08:57.521456 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e196cf1-67fe-4c0e-b2bf-9b83296e9738 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:08:57.536610 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 587/1169] 10.222.0.26 () {60 vars in 1138 bytes} [Tue Sep 27 04:08:57 2022] POST /identity/v3/domains => generated 267 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:08:59.673434 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2143e45-853e-49e1-9325-d5be161d189b None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:59.673990 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2143e45-853e-49e1-9325-d5be161d189b None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:59.673990 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2143e45-853e-49e1-9325-d5be161d189b None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:59.675239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 583/1170] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:08:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:08:59.680696 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879f9281-470b-40f8-9ce5-cf58f55d0671 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:08:59.681088 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879f9281-470b-40f8-9ce5-cf58f55d0671 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:08:59.681239 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879f9281-470b-40f8-9ce5-cf58f55d0671 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:08:59.699500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-879f9281-470b-40f8-9ce5-cf58f55d0671 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:08:59.700667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-879f9281-470b-40f8-9ce5-cf58f55d0671 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:08:59.764384 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 588/1171] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:08:59 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:09:00.031920 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2143e45-853e-49e1-9325-d5be161d189b None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:09:00.128027 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2143e45-853e-49e1-9325-d5be161d189b None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:09:00.160391 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f799258a-d0dc-4c79-845b-f295f40b090c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:09:00.160993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f799258a-d0dc-4c79-845b-f295f40b090c None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:00.161825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f799258a-d0dc-4c79-845b-f295f40b090c None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:00.162094 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f799258a-d0dc-4c79-845b-f295f40b090c None admin] PATH_INFO: `/v3/domains/70ca7802b4714dcba4a501349a51dcbb` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:00.168304 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:09:00.172637 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f799258a-d0dc-4c79-845b-f295f40b090c None admin] RBAC: Authorizing `identity:get_domain(domain_id=70ca7802b4714dcba4a501349a51dcbb)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:09:00.174071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f799258a-d0dc-4c79-845b-f295f40b090c None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:09:00.175608 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 584/1172] 10.222.0.26 () {56 vars in 1183 bytes} [Tue Sep 27 04:09:00 2022] GET /identity/v3/domains/70ca7802b4714dcba4a501349a51dcbb => generated 267 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:09:00.181327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879f9281-470b-40f8-9ce5-cf58f55d0671 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:09:00.275615 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879f9281-470b-40f8-9ce5-cf58f55d0671 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:09:00.296714 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:09:00.308904 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a254d64-5bfc-40fe-8eda-b5396b12f715 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:09:00.309496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a254d64-5bfc-40fe-8eda-b5396b12f715 None admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:00.309802 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a254d64-5bfc-40fe-8eda-b5396b12f715 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:00.310075 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a254d64-5bfc-40fe-8eda-b5396b12f715 None admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:00.310723 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a254d64-5bfc-40fe-8eda-b5396b12f715 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:09:00.312124 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a254d64-5bfc-40fe-8eda-b5396b12f715 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:09:00.337046 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 589/1173] 10.222.0.26 () {60 vars in 1141 bytes} [Tue Sep 27 04:09:00 2022] POST /identity/v3/projects => generated 361 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:09:02.461111 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4819f542-9df4-4e27-a8c7-f7b43fecf12f None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:02.461603 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4819f542-9df4-4e27-a8c7-f7b43fecf12f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:02.461715 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4819f542-9df4-4e27-a8c7-f7b43fecf12f None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:02.463008 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 585/1174] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:09:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:09:02.468155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b982b6f6-4219-455c-a559-591ad36e269e None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:02.468559 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b982b6f6-4219-455c-a559-591ad36e269e None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:02.468792 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b982b6f6-4219-455c-a559-591ad36e269e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:02.487787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-b982b6f6-4219-455c-a559-591ad36e269e None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:09:02.488944 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-b982b6f6-4219-455c-a559-591ad36e269e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:09:02.553498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 590/1175] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:09:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:09:02.815741 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4819f542-9df4-4e27-a8c7-f7b43fecf12f None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:09:02.913037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4819f542-9df4-4e27-a8c7-f7b43fecf12f None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:09:02.944386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1639a1-ba47-40ea-85e9-82b4479379fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:09:02.944984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1639a1-ba47-40ea-85e9-82b4479379fe None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:02.945253 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1639a1-ba47-40ea-85e9-82b4479379fe None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:02.945517 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1639a1-ba47-40ea-85e9-82b4479379fe None admin] PATH_INFO: `/v3/domains/70ca7802b4714dcba4a501349a51dcbb` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:02.946624 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1639a1-ba47-40ea-85e9-82b4479379fe None admin] RBAC: Authorizing `identity:get_domain(domain_id=70ca7802b4714dcba4a501349a51dcbb)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:09:02.948024 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1639a1-ba47-40ea-85e9-82b4479379fe None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:09:02.949519 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 586/1176] 10.222.0.26 () {56 vars in 1183 bytes} [Tue Sep 27 04:09:02 2022] GET /identity/v3/domains/70ca7802b4714dcba4a501349a51dcbb => generated 267 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:09:02.955072 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b982b6f6-4219-455c-a559-591ad36e269e None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:09:03.049594 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b982b6f6-4219-455c-a559-591ad36e269e None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:09:03.052343 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:09:03.084506 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b3b590-a335-4cd0-8440-3f527d07354e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:09:03.085036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b3b590-a335-4cd0-8440-3f527d07354e None admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:03.085313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b3b590-a335-4cd0-8440-3f527d07354e None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:03.085607 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b3b590-a335-4cd0-8440-3f527d07354e None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:03.086290 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b3b590-a335-4cd0-8440-3f527d07354e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:09:03.087702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b3b590-a335-4cd0-8440-3f527d07354e None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:09:03.108127 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 591/1177] 10.222.0.26 () {60 vars in 1132 bytes} [Tue Sep 27 04:09:02 2022] POST /identity/v3/users => generated 311 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:09:05.245056 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4b4da9-b1cd-4bb2-b1c4-22aa6b0b16d2 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:05.245588 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4b4da9-b1cd-4bb2-b1c4-22aa6b0b16d2 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:05.245588 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4b4da9-b1cd-4bb2-b1c4-22aa6b0b16d2 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:05.246939 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 587/1178] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:09:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:09:05.252060 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e6d730-6783-428f-bc1c-5f5028499708 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:05.252483 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e6d730-6783-428f-bc1c-5f5028499708 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:05.252595 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e6d730-6783-428f-bc1c-5f5028499708 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:05.271179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-83e6d730-6783-428f-bc1c-5f5028499708 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:09:05.272358 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-83e6d730-6783-428f-bc1c-5f5028499708 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:09:05.338689 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 592/1179] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:09:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:09:05.610776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4b4da9-b1cd-4bb2-b1c4-22aa6b0b16d2 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:09:05.673844 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:09:05.708372 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4b4da9-b1cd-4bb2-b1c4-22aa6b0b16d2 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:09:05.740340 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3312d0f5-3731-4359-a32e-c0deb4fa8f35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:09:05.740940 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3312d0f5-3731-4359-a32e-c0deb4fa8f35 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:05.741212 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3312d0f5-3731-4359-a32e-c0deb4fa8f35 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:05.741477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3312d0f5-3731-4359-a32e-c0deb4fa8f35 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:05.751418 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3312d0f5-3731-4359-a32e-c0deb4fa8f35 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:09:05.756191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-3312d0f5-3731-4359-a32e-c0deb4fa8f35 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:09:05.756191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:09:05.756191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:09:05.756191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:09:05.756191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:09:05.756191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:09:05.756191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:09:05.756191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:09:05.756191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:09:05.756191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:09:05.756191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:09:05.756191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:09:05.756191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:09:05.756191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:09:05.756191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:09:05.756191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:09:05.757554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:09:05.757554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:09:05.757554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:09:05.757554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:09:05.757554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:09:05.757554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:09:05.757554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:09:05.757554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:09:05.757554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:09:05.757554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:09:05.757554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:09:05.757554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:09:05.757554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:09:05.757554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:09:05.757554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:09:05.757554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:09:05.758786 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:09:05.758786 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:09:05.758786 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:09:05.758786 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:09:05.758786 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:09:05.758786 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:09:05.758786 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:09:05.758786 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:09:05.758786 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:09:05.758786 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:09:05.758786 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:09:05.758786 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:09:05.758786 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 588/1180] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:09:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:09:05.763544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e6d730-6783-428f-bc1c-5f5028499708 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:09:05.847065 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:09:05.857568 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e6d730-6783-428f-bc1c-5f5028499708 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:09:05.890061 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab33c8c-af3d-4fa3-910c-6a1c2b1fe350 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:09:05.890585 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab33c8c-af3d-4fa3-910c-6a1c2b1fe350 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:05.890860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab33c8c-af3d-4fa3-910c-6a1c2b1fe350 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:05.891127 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab33c8c-af3d-4fa3-910c-6a1c2b1fe350 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:05.891773 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab33c8c-af3d-4fa3-910c-6a1c2b1fe350 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:09:05.892067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab33c8c-af3d-4fa3-910c-6a1c2b1fe350 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:09:05.893403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab33c8c-af3d-4fa3-910c-6a1c2b1fe350 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:09:05.901017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 593/1181] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:09:05 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:09:05.907014 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3312d0f5-3731-4359-a32e-c0deb4fa8f35 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:09:06.003755 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3312d0f5-3731-4359-a32e-c0deb4fa8f35 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:09:06.027994 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:09:06.037442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d56b12c-e7a9-4c8e-94a5-5cbba3ba823c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:09:06.038091 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d56b12c-e7a9-4c8e-94a5-5cbba3ba823c None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:06.038375 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d56b12c-e7a9-4c8e-94a5-5cbba3ba823c None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:06.038648 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d56b12c-e7a9-4c8e-94a5-5cbba3ba823c None admin] PATH_INFO: `/v3/users/1a721f2d810547cc9232ea18e8f4d3ca` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:06.051186 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d56b12c-e7a9-4c8e-94a5-5cbba3ba823c None admin] RBAC: Authorizing `identity:get_user(user_id=1a721f2d810547cc9232ea18e8f4d3ca)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:09:06.053346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d56b12c-e7a9-4c8e-94a5-5cbba3ba823c None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:09:06.054970 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 589/1182] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:09:05 2022] GET /identity/v3/users/1a721f2d810547cc9232ea18e8f4d3ca => generated 311 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:09:06.060715 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab33c8c-af3d-4fa3-910c-6a1c2b1fe350 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:09:06.157256 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab33c8c-af3d-4fa3-910c-6a1c2b1fe350 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:09:06.188766 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6ac246-1b51-4100-9054-966654a79344 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:09:06.189346 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6ac246-1b51-4100-9054-966654a79344 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:06.189622 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6ac246-1b51-4100-9054-966654a79344 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:06.189916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6ac246-1b51-4100-9054-966654a79344 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:06.210133 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:09:06.253216 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c6ac246-1b51-4100-9054-966654a79344 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:09:06.254655 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c6ac246-1b51-4100-9054-966654a79344 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:09:06.352846 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 594/1183] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:09:06 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 04:09:06.358774 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d56b12c-e7a9-4c8e-94a5-5cbba3ba823c None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:09:06.454872 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d56b12c-e7a9-4c8e-94a5-5cbba3ba823c None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:09:06.487268 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20fe70c-9e87-4105-9b55-c5d31a95a4f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:09:06.487880 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20fe70c-9e87-4105-9b55-c5d31a95a4f9 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:06.488160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20fe70c-9e87-4105-9b55-c5d31a95a4f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:06.488435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20fe70c-9e87-4105-9b55-c5d31a95a4f9 None admin] PATH_INFO: `/v3/projects/b66d0d93f66940b1bd33a75ef2749133` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:06.489565 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b20fe70c-9e87-4105-9b55-c5d31a95a4f9 None admin] RBAC: Authorizing `identity:get_project(project_id=b66d0d93f66940b1bd33a75ef2749133)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:09:06.491008 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b20fe70c-9e87-4105-9b55-c5d31a95a4f9 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:09:06.492604 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 590/1184] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:09:06 2022] GET /identity/v3/projects/b66d0d93f66940b1bd33a75ef2749133 => generated 361 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:09:06.498654 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6ac246-1b51-4100-9054-966654a79344 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:09:06.548419 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:09:06.599450 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6ac246-1b51-4100-9054-966654a79344 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:09:06.631610 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acdcd1b0-fe7b-480f-91f4-787c200a7cae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:09:06.632448 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdcd1b0-fe7b-480f-91f4-787c200a7cae None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:06.632721 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdcd1b0-fe7b-480f-91f4-787c200a7cae None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:06.632989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdcd1b0-fe7b-480f-91f4-787c200a7cae None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:06.633693 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acdcd1b0-fe7b-480f-91f4-787c200a7cae None admin] RBAC: Adding query filter params (role.id=e6a31562a18649f0809202c36a8e18ef, scope.project.id=b66d0d93f66940b1bd33a75ef2749133, user.id=1a721f2d810547cc9232ea18e8f4d3ca) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:09:06.634015 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acdcd1b0-fe7b-480f-91f4-787c200a7cae None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:09:06.635403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acdcd1b0-fe7b-480f-91f4-787c200a7cae None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:09:06.640477 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 595/1185] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:09:06 2022] GET /identity/v3/role_assignments?user.id=1a721f2d810547cc9232ea18e8f4d3ca&scope.project.id=b66d0d93f66940b1bd33a75ef2749133&role.id=e6a31562a18649f0809202c36a8e18ef => generated 259 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:09:08.727342 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54358250-6af8-4a55-8245-be095e998ea1 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:08.727856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54358250-6af8-4a55-8245-be095e998ea1 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:08.727981 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54358250-6af8-4a55-8245-be095e998ea1 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:08.729198 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 591/1186] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:09:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:09:08.734431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ba32bc-a9d7-4015-915b-b01c7f04fa57 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:08.734808 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ba32bc-a9d7-4015-915b-b01c7f04fa57 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:08.734978 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ba32bc-a9d7-4015-915b-b01c7f04fa57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:08.753223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-95ba32bc-a9d7-4015-915b-b01c7f04fa57 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:09:08.754438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-95ba32bc-a9d7-4015-915b-b01c7f04fa57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:09:08.817895 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 596/1187] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:09:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:09:09.085135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54358250-6af8-4a55-8245-be095e998ea1 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:09:09.136505 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:09:09.182131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54358250-6af8-4a55-8245-be095e998ea1 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:09:09.213858 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34e46fe-b5d7-402a-b232-81e284e77255 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:09:09.214467 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34e46fe-b5d7-402a-b232-81e284e77255 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:09.214740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34e46fe-b5d7-402a-b232-81e284e77255 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:09.215004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34e46fe-b5d7-402a-b232-81e284e77255 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:09.225155 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34e46fe-b5d7-402a-b232-81e284e77255 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:09:09.229911 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-c34e46fe-b5d7-402a-b232-81e284e77255 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:09:09.229911 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:09:09.229911 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:09:09.229911 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:09:09.229911 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:09:09.229911 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:09:09.229911 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:09:09.229911 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:09:09.229911 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:09:09.229911 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:09:09.229911 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:09:09.229911 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:09:09.229911 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:09:09.229911 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:09:09.229911 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:09:09.229911 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:09:09.231011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:09:09.231011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:09:09.231011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:09:09.231011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:09:09.231011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:09:09.231011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:09:09.231011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:09:09.231011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:09:09.231011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:09:09.231011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:09:09.231011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:09:09.231011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:09:09.231011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:09:09.231011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:09:09.231011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:09:09.231011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:09:09.231901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:09:09.231901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:09:09.231901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:09:09.231901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:09:09.231901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:09:09.231901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:09:09.231901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:09:09.231901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:09:09.231901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:09:09.231901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:09:09.231901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:09:09.231901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:09:09.231901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 592/1188] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:09:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:09:09.236944 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ba32bc-a9d7-4015-915b-b01c7f04fa57 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:09:09.281097 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:09:09.331563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ba32bc-a9d7-4015-915b-b01c7f04fa57 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:09:09.363448 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557aeec8-aaed-46ba-b7ac-97cbd70372f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:09:09.363972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557aeec8-aaed-46ba-b7ac-97cbd70372f6 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:09.364274 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557aeec8-aaed-46ba-b7ac-97cbd70372f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:09.364543 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557aeec8-aaed-46ba-b7ac-97cbd70372f6 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:09.365188 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557aeec8-aaed-46ba-b7ac-97cbd70372f6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:09:09.365483 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557aeec8-aaed-46ba-b7ac-97cbd70372f6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:09:09.366848 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557aeec8-aaed-46ba-b7ac-97cbd70372f6 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:09:09.374425 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 597/1189] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:09:09 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:09:09.381100 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34e46fe-b5d7-402a-b232-81e284e77255 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:09:09.476672 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34e46fe-b5d7-402a-b232-81e284e77255 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:09:09.489141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:09:09.510073 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b45a98-0365-42fd-a620-e26d6cf7db01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:09:09.510686 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b45a98-0365-42fd-a620-e26d6cf7db01 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:09.510961 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b45a98-0365-42fd-a620-e26d6cf7db01 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:09.511231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b45a98-0365-42fd-a620-e26d6cf7db01 None admin] PATH_INFO: `/v3/users/1a721f2d810547cc9232ea18e8f4d3ca` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:09.512320 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b45a98-0365-42fd-a620-e26d6cf7db01 None admin] RBAC: Authorizing `identity:get_user(user_id=1a721f2d810547cc9232ea18e8f4d3ca)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:09:09.513725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b45a98-0365-42fd-a620-e26d6cf7db01 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:09:09.515295 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 593/1190] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:09:09 2022] GET /identity/v3/users/1a721f2d810547cc9232ea18e8f4d3ca => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:09:09.521625 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557aeec8-aaed-46ba-b7ac-97cbd70372f6 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:09:09.616142 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557aeec8-aaed-46ba-b7ac-97cbd70372f6 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:09:09.624481 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:09:09.649694 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeda2285-ffe5-470d-a89f-6c31ad251bbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:09:09.650334 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeda2285-ffe5-470d-a89f-6c31ad251bbd None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:09.650608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeda2285-ffe5-470d-a89f-6c31ad251bbd None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:09.650875 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeda2285-ffe5-470d-a89f-6c31ad251bbd None admin] PATH_INFO: `/v3/projects/b66d0d93f66940b1bd33a75ef2749133` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:09.651958 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeda2285-ffe5-470d-a89f-6c31ad251bbd None admin] RBAC: Authorizing `identity:get_project(project_id=b66d0d93f66940b1bd33a75ef2749133)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:09:09.653360 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeda2285-ffe5-470d-a89f-6c31ad251bbd None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:09:09.654998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 598/1191] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:09:09 2022] GET /identity/v3/projects/b66d0d93f66940b1bd33a75ef2749133 => generated 361 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:09:09.660875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b45a98-0365-42fd-a620-e26d6cf7db01 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:09:09.755806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b45a98-0365-42fd-a620-e26d6cf7db01 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:09:09.787817 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1812ca01-abd9-4786-86b4-4ec7f68f7741 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:09:09.788447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1812ca01-abd9-4786-86b4-4ec7f68f7741 None admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:09.788720 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1812ca01-abd9-4786-86b4-4ec7f68f7741 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:09.788989 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1812ca01-abd9-4786-86b4-4ec7f68f7741 None admin] PATH_INFO: `/v3/projects/b66d0d93f66940b1bd33a75ef2749133/users/1a721f2d810547cc9232ea18e8f4d3ca/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:09.790857 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1812ca01-abd9-4786-86b4-4ec7f68f7741 None admin] RBAC: Authorizing `identity:create_grant(project_id=b66d0d93f66940b1bd33a75ef2749133, user_id=1a721f2d810547cc9232ea18e8f4d3ca, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:09:09.792256 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1812ca01-abd9-4786-86b4-4ec7f68f7741 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:09:09.804721 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 594/1192] 10.222.0.26 () {58 vars in 1439 bytes} [Tue Sep 27 04:09:09 2022] PUT /identity/v3/projects/b66d0d93f66940b1bd33a75ef2749133/users/1a721f2d810547cc9232ea18e8f4d3ca/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:09:11.896372 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463ee142-2079-4540-85b4-8595052770a5 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:11.896899 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463ee142-2079-4540-85b4-8595052770a5 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:11.896899 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463ee142-2079-4540-85b4-8595052770a5 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:11.898246 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 599/1193] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:09:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:09:11.903423 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae36140c-deb6-4b6a-bb31-150eb2880dcc None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:11.903828 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae36140c-deb6-4b6a-bb31-150eb2880dcc None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:11.903960 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae36140c-deb6-4b6a-bb31-150eb2880dcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:11.921954 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-ae36140c-deb6-4b6a-bb31-150eb2880dcc None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:09:11.923105 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ae36140c-deb6-4b6a-bb31-150eb2880dcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:09:11.955355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:09:11.988376 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 595/1194] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:09:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:09:12.264823 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463ee142-2079-4540-85b4-8595052770a5 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:09:12.362893 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463ee142-2079-4540-85b4-8595052770a5 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:09:12.394887 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb612aa-83c3-4535-8a5f-1cafd99896b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:09:12.396050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb612aa-83c3-4535-8a5f-1cafd99896b0 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:12.396337 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb612aa-83c3-4535-8a5f-1cafd99896b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:12.396608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb612aa-83c3-4535-8a5f-1cafd99896b0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:12.408486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb612aa-83c3-4535-8a5f-1cafd99896b0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:09:12.413446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-bbb612aa-83c3-4535-8a5f-1cafd99896b0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:09:12.413446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:09:12.413446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:09:12.413446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:09:12.413446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:09:12.413446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:09:12.413446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:09:12.413446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:09:12.413446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:09:12.413446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:09:12.413446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:09:12.413446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:09:12.413446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:09:12.413446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:09:12.413446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:09:12.413446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:09:12.414487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:09:12.414487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:09:12.414487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:09:12.414487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:09:12.414487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:09:12.414487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:09:12.414487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:09:12.414487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:09:12.414487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:09:12.414487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:09:12.414487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:09:12.414487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:09:12.414487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:09:12.414487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:09:12.414487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:09:12.414487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:09:12.415516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:09:12.415516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:09:12.415516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:09:12.415516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:09:12.415516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:09:12.415516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:09:12.415516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:09:12.415516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:09:12.415516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:09:12.415516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:09:12.415516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:09:12.415516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:09:12.415516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 600/1195] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:09:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:09:12.420988 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae36140c-deb6-4b6a-bb31-150eb2880dcc None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:09:12.522667 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae36140c-deb6-4b6a-bb31-150eb2880dcc None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:09:12.554961 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb7193e-13d1-40a8-82f4-7d534b4a2706 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:09:12.555560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb7193e-13d1-40a8-82f4-7d534b4a2706 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:12.555834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb7193e-13d1-40a8-82f4-7d534b4a2706 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:12.556100 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb7193e-13d1-40a8-82f4-7d534b4a2706 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:12.556749 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb7193e-13d1-40a8-82f4-7d534b4a2706 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:09:12.557046 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb7193e-13d1-40a8-82f4-7d534b4a2706 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:09:12.558433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb7193e-13d1-40a8-82f4-7d534b4a2706 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:09:12.566586 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 596/1196] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:09:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:09:12.572929 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb612aa-83c3-4535-8a5f-1cafd99896b0 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:09:12.608920 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:09:12.672420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb612aa-83c3-4535-8a5f-1cafd99896b0 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:09:12.704037 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07451c3f-eda1-4185-97ee-d4b9720b6401 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:09:12.704648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07451c3f-eda1-4185-97ee-d4b9720b6401 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:12.704922 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07451c3f-eda1-4185-97ee-d4b9720b6401 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:12.705188 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07451c3f-eda1-4185-97ee-d4b9720b6401 None admin] PATH_INFO: `/v3/users/1a721f2d810547cc9232ea18e8f4d3ca` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:12.706302 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07451c3f-eda1-4185-97ee-d4b9720b6401 None admin] RBAC: Authorizing `identity:get_user(user_id=1a721f2d810547cc9232ea18e8f4d3ca)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:09:12.707707 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07451c3f-eda1-4185-97ee-d4b9720b6401 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:09:12.709890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 601/1197] 10.222.0.26 () {56 vars in 1177 bytes} [Tue Sep 27 04:09:12 2022] GET /identity/v3/users/1a721f2d810547cc9232ea18e8f4d3ca => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:09:12.715682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb7193e-13d1-40a8-82f4-7d534b4a2706 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:09:12.745821 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:09:12.811685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb7193e-13d1-40a8-82f4-7d534b4a2706 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:09:12.843487 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c585e0-b763-47d8-9435-00c5d82eb17a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:09:12.844074 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c585e0-b763-47d8-9435-00c5d82eb17a None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:12.844352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c585e0-b763-47d8-9435-00c5d82eb17a None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:12.844625 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c585e0-b763-47d8-9435-00c5d82eb17a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:12.908303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c585e0-b763-47d8-9435-00c5d82eb17a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:09:12.909728 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c585e0-b763-47d8-9435-00c5d82eb17a None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:09:12.937991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:09:13.012780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 597/1198] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:09:12 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 04:09:13.018651 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07451c3f-eda1-4185-97ee-d4b9720b6401 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:09:13.113044 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07451c3f-eda1-4185-97ee-d4b9720b6401 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:09:13.115796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:09:13.145172 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c77d7d-9054-4a5f-8742-32d760b4c197 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:09:13.145803 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c77d7d-9054-4a5f-8742-32d760b4c197 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:13.146081 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c77d7d-9054-4a5f-8742-32d760b4c197 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:13.146347 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c77d7d-9054-4a5f-8742-32d760b4c197 None admin] PATH_INFO: `/v3/projects/b66d0d93f66940b1bd33a75ef2749133` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:13.147425 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c77d7d-9054-4a5f-8742-32d760b4c197 None admin] RBAC: Authorizing `identity:get_project(project_id=b66d0d93f66940b1bd33a75ef2749133)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:09:13.148822 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c77d7d-9054-4a5f-8742-32d760b4c197 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:09:13.150432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 602/1199] 10.222.0.26 () {56 vars in 1186 bytes} [Tue Sep 27 04:09:13 2022] GET /identity/v3/projects/b66d0d93f66940b1bd33a75ef2749133 => generated 361 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:09:13.156429 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c585e0-b763-47d8-9435-00c5d82eb17a None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:09:13.251221 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c585e0-b763-47d8-9435-00c5d82eb17a None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:09:13.280179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:09:13.285245 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe005707-bb9c-4708-ab94-68d764bd710e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:09:13.285865 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe005707-bb9c-4708-ab94-68d764bd710e None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:09:13.286169 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe005707-bb9c-4708-ab94-68d764bd710e None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:09:13.286437 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe005707-bb9c-4708-ab94-68d764bd710e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:09:13.287155 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe005707-bb9c-4708-ab94-68d764bd710e None admin] RBAC: Adding query filter params (role.id=e6a31562a18649f0809202c36a8e18ef, scope.project.id=b66d0d93f66940b1bd33a75ef2749133, user.id=1a721f2d810547cc9232ea18e8f4d3ca) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:09:13.287456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe005707-bb9c-4708-ab94-68d764bd710e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:09:13.288858 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe005707-bb9c-4708-ab94-68d764bd710e None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:09:13.294497 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 598/1200] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:09:13 2022] GET /identity/v3/role_assignments?user.id=1a721f2d810547cc9232ea18e8f4d3ca&scope.project.id=b66d0d93f66940b1bd33a75ef2749133&role.id=e6a31562a18649f0809202c36a8e18ef => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:09:23.141225 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=14 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:09:23.291908 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:08.612037 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32459a8d-bad4-473c-815e-f89380be480c None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:08.612644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32459a8d-bad4-473c-815e-f89380be480c None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:08.612644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32459a8d-bad4-473c-815e-f89380be480c None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:08.614790 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 603/1201] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:10:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:10:08.620529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38a38a7-f6a0-4a41-8dfc-bfe45842266d None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:08.621079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38a38a7-f6a0-4a41-8dfc-bfe45842266d None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:08.621202 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38a38a7-f6a0-4a41-8dfc-bfe45842266d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:08.643008 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-c38a38a7-f6a0-4a41-8dfc-bfe45842266d None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:10:08.647488 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c38a38a7-f6a0-4a41-8dfc-bfe45842266d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:10:08.737786 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 599/1202] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:10:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:10:09.009811 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32459a8d-bad4-473c-815e-f89380be480c None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:09.166860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32459a8d-bad4-473c-815e-f89380be480c None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:09.205165 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447f3fe6-3ac4-453c-add1-764c768cbfd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:09.205829 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447f3fe6-3ac4-453c-add1-764c768cbfd3 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:09.206112 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447f3fe6-3ac4-453c-add1-764c768cbfd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:09.206380 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447f3fe6-3ac4-453c-add1-764c768cbfd3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:09.218131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447f3fe6-3ac4-453c-add1-764c768cbfd3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:09.221777 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447f3fe6-3ac4-453c-add1-764c768cbfd3 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:09.234954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-447f3fe6-3ac4-453c-add1-764c768cbfd3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:09.234954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:09.234954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:09.234954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:09.234954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:09.234954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:09.234954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:09.234954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:09.234954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:09.234954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:09.234954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:09.234954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:09.234954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:10:09.234954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:10:09.234954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:10:09.234954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:10:09.236187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:09.236187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:09.236187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:09.236187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:09.236187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:09.236187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:09.236187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:09.236187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:09.236187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:09.236187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:09.236187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:09.236187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:09.236187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:09.236187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:09.236187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:09.236187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:09.236187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:10:09.237356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:10:09.237356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:09.237356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:10:09.244357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 604/1203] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:10:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:09.244555 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38a38a7-f6a0-4a41-8dfc-bfe45842266d None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:09.377102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38a38a7-f6a0-4a41-8dfc-bfe45842266d None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:09.413588 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:09.418977 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8f78f7-5fd1-46e6-b6f8-f3d293ec0982 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:09.419813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8f78f7-5fd1-46e6-b6f8-f3d293ec0982 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:09.420247 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8f78f7-5fd1-46e6-b6f8-f3d293ec0982 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:09.420460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8f78f7-5fd1-46e6-b6f8-f3d293ec0982 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:09.421963 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8f78f7-5fd1-46e6-b6f8-f3d293ec0982 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:09.422225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8f78f7-5fd1-46e6-b6f8-f3d293ec0982 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:09.423647 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8f78f7-5fd1-46e6-b6f8-f3d293ec0982 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:09.436405 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 600/1204] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:10:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:10:09.443661 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447f3fe6-3ac4-453c-add1-764c768cbfd3 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:09.523131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:09.589857 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447f3fe6-3ac4-453c-add1-764c768cbfd3 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:09.629595 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f1c687-b2e7-4775-adcc-c0433a26d3aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:09.630962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f1c687-b2e7-4775-adcc-c0433a26d3aa None admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:09.630962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f1c687-b2e7-4775-adcc-c0433a26d3aa None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:09.633916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f1c687-b2e7-4775-adcc-c0433a26d3aa None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:09.633916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f1c687-b2e7-4775-adcc-c0433a26d3aa None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:09.633916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f1c687-b2e7-4775-adcc-c0433a26d3aa None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:09.661794 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 605/1205] 10.222.0.26 () {60 vars in 1132 bytes} [Tue Sep 27 04:10:09 2022] POST /identity/v3/users => generated 251 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:10:12.028535 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac726e80-126f-449b-9a7c-af5086bd6fe5 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:12.029144 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac726e80-126f-449b-9a7c-af5086bd6fe5 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:12.029250 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac726e80-126f-449b-9a7c-af5086bd6fe5 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:12.030572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 601/1206] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:10:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:10:12.040105 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed841ace-6e9d-4a79-880b-c820623b550d None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:12.040105 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed841ace-6e9d-4a79-880b-c820623b550d None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:12.040105 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed841ace-6e9d-4a79-880b-c820623b550d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:12.057270 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-ed841ace-6e9d-4a79-880b-c820623b550d None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:10:12.058892 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ed841ace-6e9d-4a79-880b-c820623b550d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:10:12.144896 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 606/1207] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:10:12 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:10:12.427128 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac726e80-126f-449b-9a7c-af5086bd6fe5 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:12.541832 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac726e80-126f-449b-9a7c-af5086bd6fe5 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:12.590389 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372450e2-b3b4-4ebe-b72f-ed08c0929a02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:12.591180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372450e2-b3b4-4ebe-b72f-ed08c0929a02 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:12.591493 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372450e2-b3b4-4ebe-b72f-ed08c0929a02 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:12.591738 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372450e2-b3b4-4ebe-b72f-ed08c0929a02 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:12.604570 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-372450e2-b3b4-4ebe-b72f-ed08c0929a02 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:10:12.610772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-372450e2-b3b4-4ebe-b72f-ed08c0929a02 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:10:12.610772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:12.610772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:12.610772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:12.610772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:12.610772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:12.610772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:12.610772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:12.610772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:12.610772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:12.610772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:12.610772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:12.610772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:10:12.610772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:10:12.610772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:10:12.610772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:10:12.611767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:10:12.611767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:10:12.611767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:10:12.611767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:10:12.611767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:12.611767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:12.611767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:12.611767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:12.611767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:12.611767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:12.611767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:12.611767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:12.611767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:12.611767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:12.611767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:12.611767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:12.613015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:12.613015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:12.613015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:12.613015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:12.613015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:10:12.613015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:10:12.613015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:10:12.613015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:10:12.613015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:10:12.613015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:10:12.613015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:10:12.613015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:10:12.613015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 602/1208] 10.222.0.26 () {56 vars in 1102 bytes} [Tue Sep 27 04:10:12 2022] GET /identity/v3/roles/service => generated 85 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:12.619348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed841ace-6e9d-4a79-880b-c820623b550d None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:12.672521 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:12.743517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed841ace-6e9d-4a79-880b-c820623b550d None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:12.781167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707ef994-31c9-4b17-9502-0beab5afd614 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:12.781811 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707ef994-31c9-4b17-9502-0beab5afd614 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:12.782097 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707ef994-31c9-4b17-9502-0beab5afd614 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:12.782373 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707ef994-31c9-4b17-9502-0beab5afd614 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:12.783048 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707ef994-31c9-4b17-9502-0beab5afd614 None admin] RBAC: Adding query filter params (name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:12.783356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707ef994-31c9-4b17-9502-0beab5afd614 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:12.784751 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707ef994-31c9-4b17-9502-0beab5afd614 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:12.798380 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 607/1209] 10.222.0.26 () {56 vars in 1103 bytes} [Tue Sep 27 04:10:12 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:12.799707 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372450e2-b3b4-4ebe-b72f-ed08c0929a02 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:12.871024 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:12.946032 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372450e2-b3b4-4ebe-b72f-ed08c0929a02 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:12.985281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bc38cd-1f8f-4610-8f2e-e6a9291ad6d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:12.985951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bc38cd-1f8f-4610-8f2e-e6a9291ad6d8 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:12.986231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bc38cd-1f8f-4610-8f2e-e6a9291ad6d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:12.986509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bc38cd-1f8f-4610-8f2e-e6a9291ad6d8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:13.007252 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00bc38cd-1f8f-4610-8f2e-e6a9291ad6d8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:13.009172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00bc38cd-1f8f-4610-8f2e-e6a9291ad6d8 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:13.028190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-00bc38cd-1f8f-4610-8f2e-e6a9291ad6d8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:13.028190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:13.028190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:13.028190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:13.028190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:13.028190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:13.028190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:13.028190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:13.028190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:13.028190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:13.028190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:13.028190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:13.028190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:10:13.028190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:10:13.028190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:10:13.028190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:10:13.029645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:13.029645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:13.029645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:13.029645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:13.029645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:13.029645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:13.029645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:13.029645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:13.029645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:13.029645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:13.029645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:13.029645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:13.029645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:13.029645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:13.029645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:13.029645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:13.029645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:10:13.030558 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:10:13.030558 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:13.030558 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:10:13.030558 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 603/1210] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:10:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:13.036334 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707ef994-31c9-4b17-9502-0beab5afd614 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:13.184591 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:13.187661 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707ef994-31c9-4b17-9502-0beab5afd614 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:13.239673 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1407e76d-e7ca-43f2-b1f8-8351dc860b7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:13.241080 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1407e76d-e7ca-43f2-b1f8-8351dc860b7a None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:13.241080 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1407e76d-e7ca-43f2-b1f8-8351dc860b7a None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:13.241294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1407e76d-e7ca-43f2-b1f8-8351dc860b7a None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:13.241881 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1407e76d-e7ca-43f2-b1f8-8351dc860b7a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:13.242186 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1407e76d-e7ca-43f2-b1f8-8351dc860b7a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:13.243761 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1407e76d-e7ca-43f2-b1f8-8351dc860b7a None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:13.261291 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 608/1211] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:10:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:13.267115 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bc38cd-1f8f-4610-8f2e-e6a9291ad6d8 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:13.376208 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bc38cd-1f8f-4610-8f2e-e6a9291ad6d8 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:13.377909 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:13.415054 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0126be99-25f6-469d-8f27-19223c76998b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:13.415691 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0126be99-25f6-469d-8f27-19223c76998b None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:13.418519 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0126be99-25f6-469d-8f27-19223c76998b None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:13.418519 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0126be99-25f6-469d-8f27-19223c76998b None admin] PATH_INFO: `/v3/users/placement` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:13.426261 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0126be99-25f6-469d-8f27-19223c76998b None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:13.427769 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0126be99-25f6-469d-8f27-19223c76998b None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:13.441456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-0126be99-25f6-469d-8f27-19223c76998b None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 27 04:10:13.441456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:13.441456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:13.441456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:13.441456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:13.441456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:13.441456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:13.441456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:13.441456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:13.441456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:13.441456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:13.441456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:13.441456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:10:13.441456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:10:13.441456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:10:13.441456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:10:13.443127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:13.443127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:13.443127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:10:13.443127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:10:13.443127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:10:13.443127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:10:13.443127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:13.443127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:13.443127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:13.443127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:13.443127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:13.443127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:13.443127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:13.443127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:13.443127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:13.443127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:13.443127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:13.443127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:13.448740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:13.448740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:13.448740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:10:13.448740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:10:13.448740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:10:13.448740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:10:13.448740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:10:13.448740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:10:13.448740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 27 04:10:13.448740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:10:13.448740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 604/1212] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:10:13 2022] GET /identity/v3/users/placement => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:10:13.450854 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1407e76d-e7ca-43f2-b1f8-8351dc860b7a None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:13.572613 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1407e76d-e7ca-43f2-b1f8-8351dc860b7a None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:13.608752 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a523daf7-7efd-41e8-b5bb-3fdc6118834d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:13.609434 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a523daf7-7efd-41e8-b5bb-3fdc6118834d None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:13.609709 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a523daf7-7efd-41e8-b5bb-3fdc6118834d None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:13.610044 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a523daf7-7efd-41e8-b5bb-3fdc6118834d None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:13.610886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a523daf7-7efd-41e8-b5bb-3fdc6118834d None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:13.611224 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a523daf7-7efd-41e8-b5bb-3fdc6118834d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:13.612745 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a523daf7-7efd-41e8-b5bb-3fdc6118834d None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:13.635008 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 609/1213] 10.222.0.26 () {56 vars in 1143 bytes} [Tue Sep 27 04:10:13 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:13.640650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0126be99-25f6-469d-8f27-19223c76998b None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:13.753258 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0126be99-25f6-469d-8f27-19223c76998b None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:13.792753 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01349fe0-18e7-463e-88b1-02179a593204 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:13.793386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01349fe0-18e7-463e-88b1-02179a593204 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:13.793653 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01349fe0-18e7-463e-88b1-02179a593204 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:13.793970 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01349fe0-18e7-463e-88b1-02179a593204 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:13.826612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:13.869541 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01349fe0-18e7-463e-88b1-02179a593204 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:13.871126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01349fe0-18e7-463e-88b1-02179a593204 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:13.983013 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 605/1214] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:10:13 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 04:10:13.989792 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a523daf7-7efd-41e8-b5bb-3fdc6118834d None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:14.011028 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:14.096333 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a523daf7-7efd-41e8-b5bb-3fdc6118834d None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:14.136805 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8487c85-891e-432c-be75-31d1360409e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:14.137452 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8487c85-891e-432c-be75-31d1360409e7 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:14.137733 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8487c85-891e-432c-be75-31d1360409e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:14.138034 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8487c85-891e-432c-be75-31d1360409e7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:14.149440 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8487c85-891e-432c-be75-31d1360409e7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:14.150990 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8487c85-891e-432c-be75-31d1360409e7 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:14.167624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-c8487c85-891e-432c-be75-31d1360409e7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:14.167624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:14.167624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:14.167624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:14.167624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:14.167624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:14.167624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:14.167624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:14.167624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:14.167624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:14.167624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:14.167624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:14.167624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:10:14.167624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:10:14.167624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:10:14.167624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:10:14.168735 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:14.168735 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:14.168735 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:14.168735 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:14.168735 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:14.168735 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:14.168735 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:14.168735 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:14.168735 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:14.168735 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:14.168735 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:14.168735 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:14.168735 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:14.168735 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:14.168735 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:14.168735 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:14.168735 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:10:14.170012 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:10:14.170012 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:14.170012 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:10:14.170012 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 610/1215] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:10:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:14.174635 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01349fe0-18e7-463e-88b1-02179a593204 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:14.254089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:14.294794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01349fe0-18e7-463e-88b1-02179a593204 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:14.334678 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7390049b-bc8e-40b0-94c1-323232e40467 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:14.335350 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7390049b-bc8e-40b0-94c1-323232e40467 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:14.335751 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7390049b-bc8e-40b0-94c1-323232e40467 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:14.336189 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7390049b-bc8e-40b0-94c1-323232e40467 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:14.337117 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7390049b-bc8e-40b0-94c1-323232e40467 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:14.337475 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7390049b-bc8e-40b0-94c1-323232e40467 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:14.339120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7390049b-bc8e-40b0-94c1-323232e40467 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:14.356585 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 606/1216] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:10:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:14.357729 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8487c85-891e-432c-be75-31d1360409e7 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:14.437054 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:14.470117 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8487c85-891e-432c-be75-31d1360409e7 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:14.512533 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e377591-2045-4187-b19a-b7dd3e2b01f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:14.513403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e377591-2045-4187-b19a-b7dd3e2b01f7 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:14.513871 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e377591-2045-4187-b19a-b7dd3e2b01f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:14.514300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e377591-2045-4187-b19a-b7dd3e2b01f7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:14.522254 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e377591-2045-4187-b19a-b7dd3e2b01f7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:14.524323 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e377591-2045-4187-b19a-b7dd3e2b01f7 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:14.540523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-7e377591-2045-4187-b19a-b7dd3e2b01f7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:10:14.540523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:14.540523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:14.540523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:14.540523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:14.540523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:14.540523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:14.540523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:14.540523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:14.540523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:14.540523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:14.540523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:14.540523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:10:14.540523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:10:14.540523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:10:14.541524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:10:14.541524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:14.541524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:14.541524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:14.541524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:14.541524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:14.541524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:14.541524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:14.541524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:14.541524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:14.541524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:14.541524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:14.541524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:14.541524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:14.541524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:14.541524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:14.541524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:14.542490 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:10:14.542490 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:10:14.542490 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:10:14.542490 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:10:14.542490 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:10:14.542490 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:10:14.542490 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:10:14.542490 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:10:14.542490 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 611/1217] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:10:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:10:14.547978 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7390049b-bc8e-40b0-94c1-323232e40467 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:14.669160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7390049b-bc8e-40b0-94c1-323232e40467 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:14.694137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:14.706905 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db6a76b-a66f-4a75-a8cf-94a83f056e74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:14.707502 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db6a76b-a66f-4a75-a8cf-94a83f056e74 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:14.707774 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db6a76b-a66f-4a75-a8cf-94a83f056e74 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:14.708099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db6a76b-a66f-4a75-a8cf-94a83f056e74 None admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:14.708805 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db6a76b-a66f-4a75-a8cf-94a83f056e74 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:14.709104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db6a76b-a66f-4a75-a8cf-94a83f056e74 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:14.710555 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db6a76b-a66f-4a75-a8cf-94a83f056e74 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:14.722751 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 607/1218] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:10:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:14.730978 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e377591-2045-4187-b19a-b7dd3e2b01f7 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:14.836107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e377591-2045-4187-b19a-b7dd3e2b01f7 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:14.865933 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:14.872031 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6842649-18f5-4649-bb04-206343eb28cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:14.872639 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6842649-18f5-4649-bb04-206343eb28cd None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:14.872906 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6842649-18f5-4649-bb04-206343eb28cd None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:14.873160 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6842649-18f5-4649-bb04-206343eb28cd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:14.873923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6842649-18f5-4649-bb04-206343eb28cd None admin] RBAC: Adding query filter params (role.id=20d83a8838004f60a32f8427577c7c35, scope.project.id=c830a3bfece6456486f4b094076b5364, user.id=3a1bc198dfc0486e9e56d4fb28cda05d) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:14.874220 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6842649-18f5-4649-bb04-206343eb28cd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:14.875614 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6842649-18f5-4649-bb04-206343eb28cd None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:14.881360 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 612/1219] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:10:14 2022] GET /identity/v3/role_assignments?user.id=3a1bc198dfc0486e9e56d4fb28cda05d&scope.project.id=c830a3bfece6456486f4b094076b5364&role.id=20d83a8838004f60a32f8427577c7c35 => generated 259 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:17.322512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8a102b-f992-4ad9-a505-b6b5335c5a5f None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:17.323063 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8a102b-f992-4ad9-a505-b6b5335c5a5f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:17.323063 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8a102b-f992-4ad9-a505-b6b5335c5a5f None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:17.324639 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 608/1220] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:10:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:10:17.331096 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da204e04-f59b-4502-84c6-2200522c4642 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:17.331604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da204e04-f59b-4502-84c6-2200522c4642 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:17.331774 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da204e04-f59b-4502-84c6-2200522c4642 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:17.352640 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-da204e04-f59b-4502-84c6-2200522c4642 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:10:17.353968 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-da204e04-f59b-4502-84c6-2200522c4642 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:10:17.432303 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 613/1221] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:10:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:10:17.740656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8a102b-f992-4ad9-a505-b6b5335c5a5f None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:17.851278 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8a102b-f992-4ad9-a505-b6b5335c5a5f None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:17.889597 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3054775a-4dca-4de2-9195-6e1975a693dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:17.890394 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3054775a-4dca-4de2-9195-6e1975a693dd None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:17.890707 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3054775a-4dca-4de2-9195-6e1975a693dd None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:17.891014 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3054775a-4dca-4de2-9195-6e1975a693dd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:17.904082 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3054775a-4dca-4de2-9195-6e1975a693dd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:10:17.909655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-3054775a-4dca-4de2-9195-6e1975a693dd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:10:17.909655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:17.909655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:17.909655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:17.909655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:17.909655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:17.909655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:17.909655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:17.909655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:17.909655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:17.909655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:17.909655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:17.909655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:10:17.909655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:10:17.909655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:10:17.909655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:10:17.910846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:10:17.910846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:10:17.910846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:10:17.910846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:10:17.910846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:17.910846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:17.910846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:17.910846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:17.910846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:17.910846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:17.910846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:17.910846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:17.910846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:17.910846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:17.910846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:17.910846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:17.911716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:17.911716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:17.911716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:17.911716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:17.911716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:10:17.911716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:10:17.911716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:10:17.911716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:10:17.911716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:10:17.911716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:10:17.911716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:10:17.911716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:10:17.911716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 609/1222] 10.222.0.26 () {56 vars in 1102 bytes} [Tue Sep 27 04:10:17 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:17.918783 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da204e04-f59b-4502-84c6-2200522c4642 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:18.032086 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da204e04-f59b-4502-84c6-2200522c4642 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:18.056040 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:18.068055 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538b0754-96de-4dfe-82c8-5d1009d83f93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:18.068652 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538b0754-96de-4dfe-82c8-5d1009d83f93 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:18.068937 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538b0754-96de-4dfe-82c8-5d1009d83f93 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:18.069200 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538b0754-96de-4dfe-82c8-5d1009d83f93 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:18.069895 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538b0754-96de-4dfe-82c8-5d1009d83f93 None admin] RBAC: Adding query filter params (name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:18.070194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538b0754-96de-4dfe-82c8-5d1009d83f93 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:18.071538 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538b0754-96de-4dfe-82c8-5d1009d83f93 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:18.080215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 614/1223] 10.222.0.26 () {56 vars in 1103 bytes} [Tue Sep 27 04:10:17 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:18.087312 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3054775a-4dca-4de2-9195-6e1975a693dd None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:18.137150 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:18.196723 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3054775a-4dca-4de2-9195-6e1975a693dd None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:18.230942 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed1cca2-5cfc-45c4-b17d-686441b15d41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:18.231624 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed1cca2-5cfc-45c4-b17d-686441b15d41 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:18.231938 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed1cca2-5cfc-45c4-b17d-686441b15d41 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:18.232209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed1cca2-5cfc-45c4-b17d-686441b15d41 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:18.242951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed1cca2-5cfc-45c4-b17d-686441b15d41 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:18.244436 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed1cca2-5cfc-45c4-b17d-686441b15d41 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:18.264138 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-6ed1cca2-5cfc-45c4-b17d-686441b15d41 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:18.264138 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:18.264138 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:18.264138 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:18.264138 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:18.264138 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:18.264138 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:18.264138 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:18.264138 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:18.264138 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:18.264138 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:18.264138 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:18.264138 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:10:18.264138 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:10:18.264138 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:10:18.264138 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:10:18.265302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:18.265302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:18.265302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:18.265302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:18.265302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:18.265302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:18.265302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:18.265302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:18.265302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:18.265302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:18.265302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:18.265302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:18.265302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:18.265302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:18.265302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:18.265302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:18.265302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:10:18.267025 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:10:18.267025 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:18.267025 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:10:18.268494 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 610/1224] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:10:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:18.274534 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538b0754-96de-4dfe-82c8-5d1009d83f93 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:18.380133 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538b0754-96de-4dfe-82c8-5d1009d83f93 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:18.419185 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8e738e-ae1f-4d53-b944-9ac0bf4cfe03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:18.419776 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8e738e-ae1f-4d53-b944-9ac0bf4cfe03 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:18.420054 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8e738e-ae1f-4d53-b944-9ac0bf4cfe03 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:18.420327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8e738e-ae1f-4d53-b944-9ac0bf4cfe03 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:18.420981 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8e738e-ae1f-4d53-b944-9ac0bf4cfe03 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:18.421277 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8e738e-ae1f-4d53-b944-9ac0bf4cfe03 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:18.422687 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8e738e-ae1f-4d53-b944-9ac0bf4cfe03 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:18.436490 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 615/1225] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:10:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:18.442768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed1cca2-5cfc-45c4-b17d-686441b15d41 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:18.541677 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:18.548929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed1cca2-5cfc-45c4-b17d-686441b15d41 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:18.586182 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd16f3d9-60b9-4c99-a6df-4dcbca8b1a32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:18.586811 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd16f3d9-60b9-4c99-a6df-4dcbca8b1a32 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:18.587085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd16f3d9-60b9-4c99-a6df-4dcbca8b1a32 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:18.587352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd16f3d9-60b9-4c99-a6df-4dcbca8b1a32 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:18.595136 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd16f3d9-60b9-4c99-a6df-4dcbca8b1a32 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:18.596558 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd16f3d9-60b9-4c99-a6df-4dcbca8b1a32 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:18.609241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-fd16f3d9-60b9-4c99-a6df-4dcbca8b1a32 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 27 04:10:18.609241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:18.609241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:18.609241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:18.609241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:18.609241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:18.609241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:18.609241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:18.609241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:18.609241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:18.609241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:18.609241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:18.609241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:10:18.609241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:10:18.609241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:10:18.609241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:10:18.610305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:18.610305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:18.610305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:10:18.610305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:10:18.610305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:10:18.610305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:10:18.610305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:18.610305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:18.610305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:18.610305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:18.610305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:18.610305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:18.610305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:18.610305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:18.610305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:18.610305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:18.610305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:18.610305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:18.611743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:18.611743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:18.611743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:10:18.611743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:10:18.611743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:10:18.611743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:10:18.611743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:10:18.611743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:10:18.611743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 27 04:10:18.611743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:10:18.611743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 611/1226] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:10:18 2022] GET /identity/v3/users/placement => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:18.616905 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8e738e-ae1f-4d53-b944-9ac0bf4cfe03 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:18.667500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:18.727938 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8e738e-ae1f-4d53-b944-9ac0bf4cfe03 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:18.762554 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748e5f47-ffbb-43b7-afb6-d31aa66b96c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:18.763156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748e5f47-ffbb-43b7-afb6-d31aa66b96c6 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:18.763431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748e5f47-ffbb-43b7-afb6-d31aa66b96c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:18.763704 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748e5f47-ffbb-43b7-afb6-d31aa66b96c6 None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:18.764424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748e5f47-ffbb-43b7-afb6-d31aa66b96c6 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:18.764726 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748e5f47-ffbb-43b7-afb6-d31aa66b96c6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:18.766173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748e5f47-ffbb-43b7-afb6-d31aa66b96c6 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:18.785176 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 616/1227] 10.222.0.26 () {56 vars in 1143 bytes} [Tue Sep 27 04:10:18 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:18.791374 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd16f3d9-60b9-4c99-a6df-4dcbca8b1a32 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:18.908570 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd16f3d9-60b9-4c99-a6df-4dcbca8b1a32 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:18.944673 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d52b758-4643-4f66-8696-adb07cb8b80b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:18.945226 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d52b758-4643-4f66-8696-adb07cb8b80b None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:18.945531 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d52b758-4643-4f66-8696-adb07cb8b80b None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:18.945808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d52b758-4643-4f66-8696-adb07cb8b80b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:18.963541 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d52b758-4643-4f66-8696-adb07cb8b80b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:18.965158 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d52b758-4643-4f66-8696-adb07cb8b80b None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:18.980126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-7d52b758-4643-4f66-8696-adb07cb8b80b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:18.980126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:18.980126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:18.980126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:18.980126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:18.980126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:18.980126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:18.980126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:18.980126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:18.980126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:18.980126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:18.980126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:18.980126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:10:18.980126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:10:18.980126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:10:18.980126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:10:18.981389 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:18.981389 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:18.981389 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:18.981389 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:18.981389 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:18.981389 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:18.981389 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:18.981389 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:18.981389 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:18.981389 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:18.981389 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:18.981389 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:18.981389 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:18.981389 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:18.981389 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:18.981389 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:18.981389 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:10:18.982658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:10:18.982658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:18.982658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:10:18.982658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 612/1228] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:10:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:18.987144 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748e5f47-ffbb-43b7-afb6-d31aa66b96c6 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:19.095523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748e5f47-ffbb-43b7-afb6-d31aa66b96c6 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:19.098248 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:19.133320 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e3ffb8-e451-467b-a286-ac14d7549f07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:19.134213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e3ffb8-e451-467b-a286-ac14d7549f07 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:19.134496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e3ffb8-e451-467b-a286-ac14d7549f07 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:19.134760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e3ffb8-e451-467b-a286-ac14d7549f07 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:19.135399 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e3ffb8-e451-467b-a286-ac14d7549f07 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:19.135696 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e3ffb8-e451-467b-a286-ac14d7549f07 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:19.137034 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e3ffb8-e451-467b-a286-ac14d7549f07 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:19.149201 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 617/1229] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:10:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:19.154961 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d52b758-4643-4f66-8696-adb07cb8b80b None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:19.164349 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:19.259702 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d52b758-4643-4f66-8696-adb07cb8b80b None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:19.296287 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9031dc0a-3d47-4e31-a0ea-b1f779e4ba2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:19.296905 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9031dc0a-3d47-4e31-a0ea-b1f779e4ba2b None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:19.297200 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9031dc0a-3d47-4e31-a0ea-b1f779e4ba2b None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:19.297469 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9031dc0a-3d47-4e31-a0ea-b1f779e4ba2b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:19.304634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9031dc0a-3d47-4e31-a0ea-b1f779e4ba2b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:19.306134 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9031dc0a-3d47-4e31-a0ea-b1f779e4ba2b None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:19.318049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-9031dc0a-3d47-4e31-a0ea-b1f779e4ba2b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:10:19.318049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:19.318049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:19.318049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:19.318049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:19.318049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:19.318049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:19.318049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:19.318049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:19.318049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:19.318049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:19.318049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:19.318049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:10:19.318049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:10:19.318049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:10:19.319126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:10:19.319126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:19.319126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:19.319126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:19.319126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:19.319126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:19.319126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:19.319126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:19.319126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:19.319126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:19.319126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:19.319126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:19.319126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:19.319126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:19.319126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:19.319126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:19.319126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:19.320400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:10:19.320400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:10:19.320400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:10:19.320400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:10:19.320400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:10:19.320400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:10:19.320400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:10:19.320400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:10:19.320400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 613/1230] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:10:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:19.325183 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e3ffb8-e451-467b-a286-ac14d7549f07 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:19.432257 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e3ffb8-e451-467b-a286-ac14d7549f07 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:19.469249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d0c096-53ec-4c83-9ce3-626508bb0591 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:19.469878 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d0c096-53ec-4c83-9ce3-626508bb0591 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:19.470153 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d0c096-53ec-4c83-9ce3-626508bb0591 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:19.470466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d0c096-53ec-4c83-9ce3-626508bb0591 None admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:19.471197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d0c096-53ec-4c83-9ce3-626508bb0591 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:19.471519 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d0c096-53ec-4c83-9ce3-626508bb0591 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:19.473012 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d0c096-53ec-4c83-9ce3-626508bb0591 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:19.486246 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 618/1231] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:10:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:19.492990 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9031dc0a-3d47-4e31-a0ea-b1f779e4ba2b None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:19.563808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:19.602923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9031dc0a-3d47-4e31-a0ea-b1f779e4ba2b None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:19.639457 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2244f0bc-e2c8-4809-b352-377cf86e313a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:19.640653 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2244f0bc-e2c8-4809-b352-377cf86e313a None admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:19.640944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2244f0bc-e2c8-4809-b352-377cf86e313a None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:19.641223 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2244f0bc-e2c8-4809-b352-377cf86e313a None admin] PATH_INFO: `/v3/projects/c830a3bfece6456486f4b094076b5364/users/3a1bc198dfc0486e9e56d4fb28cda05d/roles/20d83a8838004f60a32f8427577c7c35` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:19.656483 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2244f0bc-e2c8-4809-b352-377cf86e313a None admin] RBAC: Authorizing `identity:create_grant(project_id=c830a3bfece6456486f4b094076b5364, user_id=3a1bc198dfc0486e9e56d4fb28cda05d, role_id=20d83a8838004f60a32f8427577c7c35)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:19.657962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2244f0bc-e2c8-4809-b352-377cf86e313a None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:19.669923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 614/1232] 10.222.0.26 () {58 vars in 1439 bytes} [Tue Sep 27 04:10:19 2022] PUT /identity/v3/projects/c830a3bfece6456486f4b094076b5364/users/3a1bc198dfc0486e9e56d4fb28cda05d/roles/20d83a8838004f60a32f8427577c7c35 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:10:21.888513 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3665a63b-c15b-4e44-b51b-12e925cb64dc None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:21.889171 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3665a63b-c15b-4e44-b51b-12e925cb64dc None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:21.889171 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3665a63b-c15b-4e44-b51b-12e925cb64dc None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:21.890456 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 619/1233] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:10:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:10:21.896601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb06bbc-0765-48da-a2b4-a5f929dd827d None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:21.897044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb06bbc-0765-48da-a2b4-a5f929dd827d None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:21.897130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb06bbc-0765-48da-a2b4-a5f929dd827d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:21.917594 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-edb06bbc-0765-48da-a2b4-a5f929dd827d None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:10:21.918873 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-edb06bbc-0765-48da-a2b4-a5f929dd827d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:10:21.988352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 615/1234] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:10:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 04:10:22.298285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3665a63b-c15b-4e44-b51b-12e925cb64dc None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:22.325952 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:22.410927 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3665a63b-c15b-4e44-b51b-12e925cb64dc None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:22.451058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4606e14c-2a10-4c5f-9684-30361d993c7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:22.451723 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4606e14c-2a10-4c5f-9684-30361d993c7f None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:22.452005 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4606e14c-2a10-4c5f-9684-30361d993c7f None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:22.452418 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4606e14c-2a10-4c5f-9684-30361d993c7f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:22.466229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4606e14c-2a10-4c5f-9684-30361d993c7f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:10:22.471749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-4606e14c-2a10-4c5f-9684-30361d993c7f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:10:22.471749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:22.471749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:22.471749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:22.471749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:22.471749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:22.471749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:22.471749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:22.471749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:22.471749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:22.471749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:22.471749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:22.471749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:10:22.471749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:10:22.471749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:10:22.471749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:10:22.474354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:10:22.474354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:10:22.474354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:10:22.474354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:10:22.474354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:22.474354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:22.474354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:22.474354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:22.474354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:22.474354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:22.474354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:22.474354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:22.474354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:22.474354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:22.474354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:22.474354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:22.475282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:22.475282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:22.475282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:22.475282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:22.475282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:10:22.475282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:10:22.475282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:10:22.475282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:10:22.475282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:10:22.475282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:10:22.475282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 04:10:22.475282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:10:22.475282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 620/1235] 10.222.0.26 () {56 vars in 1102 bytes} [Tue Sep 27 04:10:22 2022] GET /identity/v3/roles/service => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:22.479179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb06bbc-0765-48da-a2b4-a5f929dd827d None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:22.527461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:22.583262 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb06bbc-0765-48da-a2b4-a5f929dd827d None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:22.618306 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb294d4-26f4-40a9-a0dc-b63b5d8d23a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:22.618913 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb294d4-26f4-40a9-a0dc-b63b5d8d23a9 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:22.619210 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb294d4-26f4-40a9-a0dc-b63b5d8d23a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:22.619459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb294d4-26f4-40a9-a0dc-b63b5d8d23a9 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:22.620123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb294d4-26f4-40a9-a0dc-b63b5d8d23a9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:22.620418 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb294d4-26f4-40a9-a0dc-b63b5d8d23a9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:22.621784 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb294d4-26f4-40a9-a0dc-b63b5d8d23a9 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:22.629685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 616/1236] 10.222.0.26 () {56 vars in 1103 bytes} [Tue Sep 27 04:10:22 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:22.636818 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4606e14c-2a10-4c5f-9684-30361d993c7f None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:22.725176 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:22.741701 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4606e14c-2a10-4c5f-9684-30361d993c7f None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:22.777230 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340b9331-c0eb-40d4-bd0e-4eead640731e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:22.777890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340b9331-c0eb-40d4-bd0e-4eead640731e None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:22.778186 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340b9331-c0eb-40d4-bd0e-4eead640731e None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:22.778444 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340b9331-c0eb-40d4-bd0e-4eead640731e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:22.788773 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340b9331-c0eb-40d4-bd0e-4eead640731e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:22.790275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340b9331-c0eb-40d4-bd0e-4eead640731e None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:22.805787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-340b9331-c0eb-40d4-bd0e-4eead640731e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:22.805787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:22.805787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:22.805787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:22.805787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:22.805787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:22.805787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:22.805787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:22.805787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:22.805787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:22.805787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:22.805787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:22.805787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:10:22.805787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:10:22.805787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:10:22.805787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:10:22.807019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:22.807019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:22.807019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:22.807019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:22.807019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:22.807019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:22.807019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:22.807019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:22.807019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:22.807019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:22.807019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:22.807019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:22.807019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:22.807019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:22.807019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:22.807019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:22.807019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:10:22.808155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:10:22.808155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:22.808155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:10:22.808155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 621/1237] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:10:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:22.812910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb294d4-26f4-40a9-a0dc-b63b5d8d23a9 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:22.923698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb294d4-26f4-40a9-a0dc-b63b5d8d23a9 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:22.935416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:22.968239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96340b7c-c4aa-45cd-ab76-3ae13418e204 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:22.969044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96340b7c-c4aa-45cd-ab76-3ae13418e204 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:22.969326 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96340b7c-c4aa-45cd-ab76-3ae13418e204 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:22.969598 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96340b7c-c4aa-45cd-ab76-3ae13418e204 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:22.970404 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96340b7c-c4aa-45cd-ab76-3ae13418e204 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:22.970805 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96340b7c-c4aa-45cd-ab76-3ae13418e204 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:22.972700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96340b7c-c4aa-45cd-ab76-3ae13418e204 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:22.986509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 617/1238] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:10:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:22.992989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340b9331-c0eb-40d4-bd0e-4eead640731e None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:23.112282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340b9331-c0eb-40d4-bd0e-4eead640731e None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:23.156233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2539a41-202a-4164-ace7-09e15a2e3c6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:23.157118 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2539a41-202a-4164-ace7-09e15a2e3c6e None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:23.157400 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2539a41-202a-4164-ace7-09e15a2e3c6e None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:23.157672 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2539a41-202a-4164-ace7-09e15a2e3c6e None admin] PATH_INFO: `/v3/users/placement` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:23.162015 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:23.169515 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2539a41-202a-4164-ace7-09e15a2e3c6e None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:23.171220 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2539a41-202a-4164-ace7-09e15a2e3c6e None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:23.183851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-f2539a41-202a-4164-ace7-09e15a2e3c6e None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 27 04:10:23.183851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:23.183851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:23.183851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:23.183851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:23.183851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:23.183851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:23.183851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:23.183851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:23.183851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:23.183851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:23.183851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:23.183851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:10:23.183851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:10:23.183851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:10:23.183851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:10:23.185157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:23.185157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:23.185157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:10:23.185157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:10:23.185157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:10:23.185157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:10:23.185157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:23.185157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:23.185157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:23.185157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:23.185157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:23.185157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:23.185157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:23.185157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:23.185157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:23.185157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:23.185157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:23.185157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:23.186203 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:23.186203 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:23.186203 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:10:23.186203 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:10:23.186203 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:10:23.186203 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:10:23.186203 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:10:23.186203 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:10:23.186203 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 27 04:10:23.186203 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:10:23.186203 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 622/1239] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:10:22 2022] GET /identity/v3/users/placement => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:23.199632 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96340b7c-c4aa-45cd-ab76-3ae13418e204 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:23.316316 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96340b7c-c4aa-45cd-ab76-3ae13418e204 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:23.354455 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397c738f-db4e-4774-ab71-826abe996542 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:23.354984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397c738f-db4e-4774-ab71-826abe996542 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:23.355265 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397c738f-db4e-4774-ab71-826abe996542 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:23.355572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397c738f-db4e-4774-ab71-826abe996542 None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:23.356339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397c738f-db4e-4774-ab71-826abe996542 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:23.356686 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397c738f-db4e-4774-ab71-826abe996542 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:23.358129 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397c738f-db4e-4774-ab71-826abe996542 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:23.377583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 618/1240] 10.222.0.26 () {56 vars in 1143 bytes} [Tue Sep 27 04:10:23 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:23.384546 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2539a41-202a-4164-ace7-09e15a2e3c6e None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:23.499615 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2539a41-202a-4164-ace7-09e15a2e3c6e None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:23.538148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad7958a-62db-4f4c-8a50-43a3f5b9e5a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:23.538840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad7958a-62db-4f4c-8a50-43a3f5b9e5a6 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:23.539227 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad7958a-62db-4f4c-8a50-43a3f5b9e5a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:23.539538 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad7958a-62db-4f4c-8a50-43a3f5b9e5a6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:23.605506 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:23.612722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad7958a-62db-4f4c-8a50-43a3f5b9e5a6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:23.614797 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad7958a-62db-4f4c-8a50-43a3f5b9e5a6 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:23.731372 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 623/1241] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:10:23 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 04:10:23.737487 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397c738f-db4e-4774-ab71-826abe996542 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:23.767714 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:23.845601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397c738f-db4e-4774-ab71-826abe996542 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:23.883803 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4765de-686c-4b8d-ace1-1d7d5b741665 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:23.884792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4765de-686c-4b8d-ace1-1d7d5b741665 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:23.885132 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4765de-686c-4b8d-ace1-1d7d5b741665 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:23.885438 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4765de-686c-4b8d-ace1-1d7d5b741665 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:23.896180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4765de-686c-4b8d-ace1-1d7d5b741665 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:23.897585 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4765de-686c-4b8d-ace1-1d7d5b741665 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:23.912534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-ef4765de-686c-4b8d-ace1-1d7d5b741665 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:23.912534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:23.912534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:23.912534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:23.912534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:23.912534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:23.912534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:23.912534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:23.912534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:23.912534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:23.912534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:23.912534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:23.912534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:10:23.912534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:10:23.912534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:10:23.912534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:10:23.913701 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:23.913701 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:23.913701 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:23.913701 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:23.913701 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:23.913701 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:23.913701 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:23.913701 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:23.913701 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:23.913701 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:23.913701 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:23.913701 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:23.913701 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:23.913701 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:23.913701 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:23.913701 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:23.913701 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:10:23.914641 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:10:23.914641 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:23.914641 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:10:23.914641 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 619/1242] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:10:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:10:23.919489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad7958a-62db-4f4c-8a50-43a3f5b9e5a6 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:24.026446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:24.032871 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad7958a-62db-4f4c-8a50-43a3f5b9e5a6 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:24.068920 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664c17bc-dab2-4f18-931d-ead036739b80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:24.069477 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664c17bc-dab2-4f18-931d-ead036739b80 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:24.069775 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664c17bc-dab2-4f18-931d-ead036739b80 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:24.070051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664c17bc-dab2-4f18-931d-ead036739b80 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:24.070705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664c17bc-dab2-4f18-931d-ead036739b80 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:24.071001 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664c17bc-dab2-4f18-931d-ead036739b80 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:24.072408 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664c17bc-dab2-4f18-931d-ead036739b80 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:24.084663 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 624/1243] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:10:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:24.090386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4765de-686c-4b8d-ace1-1d7d5b741665 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:24.181839 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:24.205202 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4765de-686c-4b8d-ace1-1d7d5b741665 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:24.243369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a67c0d-28ac-4400-b9b0-00f16b4b9bb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:24.243994 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a67c0d-28ac-4400-b9b0-00f16b4b9bb0 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:24.244285 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a67c0d-28ac-4400-b9b0-00f16b4b9bb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:24.244540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a67c0d-28ac-4400-b9b0-00f16b4b9bb0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:24.251555 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a67c0d-28ac-4400-b9b0-00f16b4b9bb0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:24.253273 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a67c0d-28ac-4400-b9b0-00f16b4b9bb0 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:24.266795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-04a67c0d-28ac-4400-b9b0-00f16b4b9bb0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:10:24.266795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:24.266795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:24.266795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:24.266795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:24.266795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:24.266795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:24.266795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:24.266795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:24.266795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:24.266795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:24.266795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:24.266795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:10:24.266795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:10:24.266795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:10:24.267993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:10:24.267993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:24.267993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:24.267993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:24.267993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:24.267993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:24.267993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:24.267993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:24.267993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:24.267993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:24.267993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:24.267993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:24.267993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:24.267993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:24.267993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:24.267993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:24.267993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:24.269403 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:10:24.269403 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:10:24.269403 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:10:24.269403 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:10:24.269403 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:10:24.269403 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:10:24.269403 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:10:24.269403 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:10:24.269403 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 620/1244] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:10:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:24.274437 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664c17bc-dab2-4f18-931d-ead036739b80 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:24.383136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664c17bc-dab2-4f18-931d-ead036739b80 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:24.431300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a817721-70f1-4f36-b077-0bd541c485a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:24.431911 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a817721-70f1-4f36-b077-0bd541c485a6 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:24.432210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a817721-70f1-4f36-b077-0bd541c485a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:24.432463 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a817721-70f1-4f36-b077-0bd541c485a6 None admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:24.433264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a817721-70f1-4f36-b077-0bd541c485a6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:24.433570 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a817721-70f1-4f36-b077-0bd541c485a6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:24.435100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a817721-70f1-4f36-b077-0bd541c485a6 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:24.453363 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 625/1245] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:10:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:24.459914 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a67c0d-28ac-4400-b9b0-00f16b4b9bb0 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:24.578195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a67c0d-28ac-4400-b9b0-00f16b4b9bb0 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:24.618055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f2590c-e613-4989-a4ee-2f272fe70fce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:24.618651 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f2590c-e613-4989-a4ee-2f272fe70fce None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:24.618961 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f2590c-e613-4989-a4ee-2f272fe70fce None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:24.619194 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f2590c-e613-4989-a4ee-2f272fe70fce None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:24.620188 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f2590c-e613-4989-a4ee-2f272fe70fce None admin] RBAC: Adding query filter params (role.id=20d83a8838004f60a32f8427577c7c35, scope.project.id=c830a3bfece6456486f4b094076b5364, user.id=3a1bc198dfc0486e9e56d4fb28cda05d) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:24.620582 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f2590c-e613-4989-a4ee-2f272fe70fce None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:24.622305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f2590c-e613-4989-a4ee-2f272fe70fce None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:24.629105 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 621/1246] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:10:24 2022] GET /identity/v3/role_assignments?user.id=3a1bc198dfc0486e9e56d4fb28cda05d&scope.project.id=c830a3bfece6456486f4b094076b5364&role.id=20d83a8838004f60a32f8427577c7c35 => generated 608 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:27.198727 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9b6d98-fb05-4641-a914-7d0cdb395d2f None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:27.199290 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9b6d98-fb05-4641-a914-7d0cdb395d2f None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:27.199290 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9b6d98-fb05-4641-a914-7d0cdb395d2f None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:27.200573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 626/1247] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:10:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:10:27.205889 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ab2e77-2bc6-4cdb-a50e-47392afd1205 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:27.206308 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ab2e77-2bc6-4cdb-a50e-47392afd1205 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:27.206466 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ab2e77-2bc6-4cdb-a50e-47392afd1205 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:27.212724 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:27.225323 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-60ab2e77-2bc6-4cdb-a50e-47392afd1205 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:10:27.226663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-60ab2e77-2bc6-4cdb-a50e-47392afd1205 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:10:27.300852 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 622/1248] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:10:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:10:27.604392 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9b6d98-fb05-4641-a914-7d0cdb395d2f None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:27.624792 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:27.719027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9b6d98-fb05-4641-a914-7d0cdb395d2f None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:27.754475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4448d2c5-d081-4b84-85c2-be63bd1b4b95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:27.755094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4448d2c5-d081-4b84-85c2-be63bd1b4b95 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:27.755373 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4448d2c5-d081-4b84-85c2-be63bd1b4b95 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:27.755664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4448d2c5-d081-4b84-85c2-be63bd1b4b95 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:27.767142 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4448d2c5-d081-4b84-85c2-be63bd1b4b95 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:10:27.772664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-4448d2c5-d081-4b84-85c2-be63bd1b4b95 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:10:27.772664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:27.772664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:27.772664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:27.772664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:27.772664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:27.772664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:27.772664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:27.772664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:27.772664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:27.772664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:27.772664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:27.772664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:10:27.772664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:10:27.772664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:10:27.772664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:10:27.773629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:10:27.773629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:10:27.773629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:10:27.773629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:10:27.773629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:27.773629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:27.773629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:27.773629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:27.773629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:27.773629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:27.773629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:27.773629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:27.773629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:27.773629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:27.773629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:27.773629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:27.774590 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:27.774590 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:27.774590 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:27.774590 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:27.774590 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:10:27.774590 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:10:27.774590 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:10:27.774590 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:10:27.774590 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:10:27.774590 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:10:27.774590 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:10:27.774590 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:10:27.774590 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 627/1249] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:10:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:27.780365 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ab2e77-2bc6-4cdb-a50e-47392afd1205 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:27.884489 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ab2e77-2bc6-4cdb-a50e-47392afd1205 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:27.915918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:27.922198 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baade230-1c70-41a5-9800-a6253ba02a31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:27.922926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baade230-1c70-41a5-9800-a6253ba02a31 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:27.923206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baade230-1c70-41a5-9800-a6253ba02a31 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:27.923609 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baade230-1c70-41a5-9800-a6253ba02a31 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:27.924471 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baade230-1c70-41a5-9800-a6253ba02a31 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:27.924777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baade230-1c70-41a5-9800-a6253ba02a31 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:27.926503 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baade230-1c70-41a5-9800-a6253ba02a31 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:27.936366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 623/1250] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:10:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:27.943286 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4448d2c5-d081-4b84-85c2-be63bd1b4b95 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:28.022298 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:28.045972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4448d2c5-d081-4b84-85c2-be63bd1b4b95 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:28.082412 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4247c8-dfb5-44cf-afe3-e80253e1a5f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:28.083619 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4247c8-dfb5-44cf-afe3-e80253e1a5f4 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:28.083900 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4247c8-dfb5-44cf-afe3-e80253e1a5f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:28.084171 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4247c8-dfb5-44cf-afe3-e80253e1a5f4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:28.095349 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4247c8-dfb5-44cf-afe3-e80253e1a5f4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:28.096796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4247c8-dfb5-44cf-afe3-e80253e1a5f4 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:28.113036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-7a4247c8-dfb5-44cf-afe3-e80253e1a5f4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:28.113036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:28.113036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:28.113036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:28.113036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:28.113036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:28.113036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:28.113036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:28.113036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:28.113036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:28.113036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:28.113036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:28.113036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:10:28.113036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:10:28.113036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:10:28.113036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:10:28.114325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:28.114325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:28.114325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:28.114325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:28.114325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:28.114325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:28.114325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:28.114325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:28.114325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:28.114325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:28.114325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:28.114325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:28.114325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:28.114325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:28.114325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:28.114325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:28.114325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:10:28.115229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:10:28.115229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:28.115229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:10:28.115229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 628/1251] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:10:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:28.121157 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baade230-1c70-41a5-9800-a6253ba02a31 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:28.230679 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baade230-1c70-41a5-9800-a6253ba02a31 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:28.266025 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad29183-a77a-45b3-9b6b-a6f812ca6044 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:28.266624 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad29183-a77a-45b3-9b6b-a6f812ca6044 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:28.266897 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad29183-a77a-45b3-9b6b-a6f812ca6044 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:28.267165 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad29183-a77a-45b3-9b6b-a6f812ca6044 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:28.267796 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad29183-a77a-45b3-9b6b-a6f812ca6044 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:28.268092 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad29183-a77a-45b3-9b6b-a6f812ca6044 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:28.269449 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad29183-a77a-45b3-9b6b-a6f812ca6044 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:28.282822 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 624/1252] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:10:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:28.288853 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4247c8-dfb5-44cf-afe3-e80253e1a5f4 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:28.392130 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4247c8-dfb5-44cf-afe3-e80253e1a5f4 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:28.421414 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:28.427732 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ca60c1-d870-4d31-a3d9-4e56796eb8fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:28.428375 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ca60c1-d870-4d31-a3d9-4e56796eb8fe None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:28.428652 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ca60c1-d870-4d31-a3d9-4e56796eb8fe None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:28.428921 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ca60c1-d870-4d31-a3d9-4e56796eb8fe None admin] PATH_INFO: `/v3/users/placement` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:28.437419 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ca60c1-d870-4d31-a3d9-4e56796eb8fe None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:28.438921 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ca60c1-d870-4d31-a3d9-4e56796eb8fe None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:28.451432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-54ca60c1-d870-4d31-a3d9-4e56796eb8fe None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 27 04:10:28.451432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:28.451432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:28.451432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:28.451432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:28.451432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:28.451432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:28.451432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:28.451432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:28.451432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:28.451432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:28.451432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:28.451432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:10:28.451432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:10:28.451432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:10:28.451432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:10:28.452644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:28.452644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:28.452644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:10:28.452644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:10:28.452644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:10:28.452644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:10:28.452644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:28.452644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:28.452644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:28.452644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:28.452644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:28.452644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:28.452644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:28.452644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:28.452644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:28.452644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:28.452644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:28.452644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:28.454000 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:28.454000 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:28.454000 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:10:28.454000 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:10:28.454000 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:10:28.454000 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:10:28.454000 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:10:28.454000 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:10:28.454000 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 27 04:10:28.454000 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:10:28.454000 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 629/1253] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:10:28 2022] GET /identity/v3/users/placement => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:28.459096 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad29183-a77a-45b3-9b6b-a6f812ca6044 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:28.518068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:28.564977 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad29183-a77a-45b3-9b6b-a6f812ca6044 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:28.598771 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71def12-55aa-4efb-a087-631d317de94e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:28.599362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71def12-55aa-4efb-a087-631d317de94e None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:28.599632 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71def12-55aa-4efb-a087-631d317de94e None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:28.599897 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71def12-55aa-4efb-a087-631d317de94e None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:28.600640 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71def12-55aa-4efb-a087-631d317de94e None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:28.600942 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71def12-55aa-4efb-a087-631d317de94e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:28.602379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71def12-55aa-4efb-a087-631d317de94e None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:28.619968 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 625/1254] 10.222.0.26 () {56 vars in 1143 bytes} [Tue Sep 27 04:10:28 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:28.625659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ca60c1-d870-4d31-a3d9-4e56796eb8fe None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:28.726266 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ca60c1-d870-4d31-a3d9-4e56796eb8fe None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:28.761067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c66470-992c-4c58-b292-77ad2279fd35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:28.761985 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c66470-992c-4c58-b292-77ad2279fd35 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:28.762423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c66470-992c-4c58-b292-77ad2279fd35 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:28.762692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c66470-992c-4c58-b292-77ad2279fd35 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:28.819487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:28.831771 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c66470-992c-4c58-b292-77ad2279fd35 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:28.833233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c66470-992c-4c58-b292-77ad2279fd35 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:28.927552 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 630/1255] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:10:28 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 04:10:28.933398 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71def12-55aa-4efb-a087-631d317de94e None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:29.028611 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71def12-55aa-4efb-a087-631d317de94e None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:29.041856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:29.062909 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967a5bb8-b093-4e84-9a60-2fa36f4920c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:29.063526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967a5bb8-b093-4e84-9a60-2fa36f4920c7 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:29.063805 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967a5bb8-b093-4e84-9a60-2fa36f4920c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:29.064076 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967a5bb8-b093-4e84-9a60-2fa36f4920c7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:29.073764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967a5bb8-b093-4e84-9a60-2fa36f4920c7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:29.075139 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967a5bb8-b093-4e84-9a60-2fa36f4920c7 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:29.091072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-967a5bb8-b093-4e84-9a60-2fa36f4920c7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:29.091072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:29.091072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:29.091072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:29.091072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:29.091072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:29.091072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:29.091072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:29.091072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:29.091072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:29.091072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:29.091072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:29.091072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:10:29.091072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:10:29.091072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:10:29.091072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:10:29.092127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:29.092127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:29.092127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:29.092127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:29.092127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:29.092127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:29.092127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:29.092127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:29.092127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:29.092127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:29.092127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:29.092127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:29.092127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:29.092127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:29.092127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:29.092127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:29.092127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:10:29.093130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:10:29.093130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:29.093130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:10:29.093130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 626/1256] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:10:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:29.099467 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c66470-992c-4c58-b292-77ad2279fd35 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:29.189935 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:29.201612 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c66470-992c-4c58-b292-77ad2279fd35 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:29.234621 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e3ceb6-73b5-4a6d-958f-cbe4a690a048 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:29.235214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e3ceb6-73b5-4a6d-958f-cbe4a690a048 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:29.235492 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e3ceb6-73b5-4a6d-958f-cbe4a690a048 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:29.235760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e3ceb6-73b5-4a6d-958f-cbe4a690a048 None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:29.236393 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e3ceb6-73b5-4a6d-958f-cbe4a690a048 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:29.236691 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e3ceb6-73b5-4a6d-958f-cbe4a690a048 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:29.238058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e3ceb6-73b5-4a6d-958f-cbe4a690a048 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:29.248778 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 631/1257] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:10:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:10:29.254521 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967a5bb8-b093-4e84-9a60-2fa36f4920c7 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:29.350639 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967a5bb8-b093-4e84-9a60-2fa36f4920c7 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:29.382137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1122c22-56fa-4a27-8077-cf3b1f890afc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:29.382746 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1122c22-56fa-4a27-8077-cf3b1f890afc None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:29.383025 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1122c22-56fa-4a27-8077-cf3b1f890afc None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:29.383298 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1122c22-56fa-4a27-8077-cf3b1f890afc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:29.389958 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1122c22-56fa-4a27-8077-cf3b1f890afc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:29.391374 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1122c22-56fa-4a27-8077-cf3b1f890afc None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:29.402794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-c1122c22-56fa-4a27-8077-cf3b1f890afc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:10:29.402794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:29.402794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:29.402794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:29.402794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:29.402794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:29.402794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:29.402794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:29.402794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:29.402794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:29.402794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:29.402794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:29.402794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:10:29.402794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:10:29.402794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:10:29.403918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:10:29.403918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:29.403918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:29.403918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:29.403918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:29.403918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:29.403918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:29.403918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:29.403918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:29.403918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:29.403918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:29.403918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:29.403918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:29.403918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:29.403918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:29.403918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:29.403918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:29.405185 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:10:29.405185 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:10:29.405185 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:10:29.405185 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:10:29.405185 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:10:29.405185 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:10:29.405185 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:10:29.405185 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:10:29.405185 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 627/1258] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:10:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:29.409707 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e3ceb6-73b5-4a6d-958f-cbe4a690a048 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:29.505857 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e3ceb6-73b5-4a6d-958f-cbe4a690a048 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:29.538705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f19e759-aa21-4809-9465-862f68cfb252 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:29.539294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f19e759-aa21-4809-9465-862f68cfb252 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:29.539575 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f19e759-aa21-4809-9465-862f68cfb252 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:29.539850 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f19e759-aa21-4809-9465-862f68cfb252 None admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:29.540533 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f19e759-aa21-4809-9465-862f68cfb252 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:29.540838 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f19e759-aa21-4809-9465-862f68cfb252 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:29.542278 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f19e759-aa21-4809-9465-862f68cfb252 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:29.553718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 632/1259] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:10:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:29.559644 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1122c22-56fa-4a27-8077-cf3b1f890afc None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:29.585335 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:29.657102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1122c22-56fa-4a27-8077-cf3b1f890afc None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:29.688367 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856e1d21-407a-4255-a21b-65959242f419 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:29.688945 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856e1d21-407a-4255-a21b-65959242f419 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:29.689225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856e1d21-407a-4255-a21b-65959242f419 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:29.689495 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856e1d21-407a-4255-a21b-65959242f419 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:29.690236 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856e1d21-407a-4255-a21b-65959242f419 None admin] RBAC: Adding query filter params (role.id=e6a31562a18649f0809202c36a8e18ef, scope.project.id=c830a3bfece6456486f4b094076b5364, user.id=3a1bc198dfc0486e9e56d4fb28cda05d) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:29.690545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856e1d21-407a-4255-a21b-65959242f419 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:29.691946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856e1d21-407a-4255-a21b-65959242f419 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:29.697593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 628/1260] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:10:29 2022] GET /identity/v3/role_assignments?user.id=3a1bc198dfc0486e9e56d4fb28cda05d&scope.project.id=c830a3bfece6456486f4b094076b5364&role.id=e6a31562a18649f0809202c36a8e18ef => generated 259 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:31.828480 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d58740-dc9d-4ee3-8963-50d9d0a96c02 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:31.829011 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d58740-dc9d-4ee3-8963-50d9d0a96c02 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:31.829011 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d58740-dc9d-4ee3-8963-50d9d0a96c02 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:31.830376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 633/1261] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:10:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:10:31.835438 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e65830-b880-4ac8-ba72-88a1fe655c1b None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:31.835811 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e65830-b880-4ac8-ba72-88a1fe655c1b None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:31.835976 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e65830-b880-4ac8-ba72-88a1fe655c1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:31.854061 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-81e65830-b880-4ac8-ba72-88a1fe655c1b None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:10:31.855208 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-81e65830-b880-4ac8-ba72-88a1fe655c1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:10:31.919535 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 629/1262] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:10:31 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:10:32.184137 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d58740-dc9d-4ee3-8963-50d9d0a96c02 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:32.194933 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:32.283218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d58740-dc9d-4ee3-8963-50d9d0a96c02 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:32.316925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2557923-1f8d-4fe9-90c7-f780e30a3268 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:32.317681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2557923-1f8d-4fe9-90c7-f780e30a3268 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:32.317990 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2557923-1f8d-4fe9-90c7-f780e30a3268 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:32.318262 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2557923-1f8d-4fe9-90c7-f780e30a3268 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:32.329829 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2557923-1f8d-4fe9-90c7-f780e30a3268 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:10:32.335242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-f2557923-1f8d-4fe9-90c7-f780e30a3268 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:10:32.335242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:32.335242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:32.335242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:32.335242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:32.335242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:32.335242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:32.335242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:32.335242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:32.335242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:32.335242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:32.335242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:32.335242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:10:32.335242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:10:32.335242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:10:32.335242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:10:32.336344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:10:32.336344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:10:32.336344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:10:32.336344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:10:32.336344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:32.336344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:32.336344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:32.336344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:32.336344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:32.336344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:32.336344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:32.336344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:32.336344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:32.336344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:32.336344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:32.336344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:32.337557 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:32.337557 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:32.337557 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:32.337557 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:32.337557 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:10:32.337557 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:10:32.337557 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:10:32.337557 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:10:32.337557 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:10:32.337557 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:10:32.337557 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:10:32.337557 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:10:32.337557 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 634/1263] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:10:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:32.342363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e65830-b880-4ac8-ba72-88a1fe655c1b None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:32.360369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:32.438512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e65830-b880-4ac8-ba72-88a1fe655c1b None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:32.470994 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e54c31-d598-47f3-8cc1-6ff0320770e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:32.471590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e54c31-d598-47f3-8cc1-6ff0320770e2 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:32.471868 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e54c31-d598-47f3-8cc1-6ff0320770e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:32.472156 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e54c31-d598-47f3-8cc1-6ff0320770e2 None admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:32.472789 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e54c31-d598-47f3-8cc1-6ff0320770e2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:32.473087 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e54c31-d598-47f3-8cc1-6ff0320770e2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:32.475076 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e54c31-d598-47f3-8cc1-6ff0320770e2 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:32.482777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 630/1264] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:10:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:32.488541 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2557923-1f8d-4fe9-90c7-f780e30a3268 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:32.553239 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:32.585627 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2557923-1f8d-4fe9-90c7-f780e30a3268 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:32.619443 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdabc647-8c4c-462b-ba28-49edc3b02f43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:32.620052 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdabc647-8c4c-462b-ba28-49edc3b02f43 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:32.620332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdabc647-8c4c-462b-ba28-49edc3b02f43 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:32.620601 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdabc647-8c4c-462b-ba28-49edc3b02f43 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:32.630712 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdabc647-8c4c-462b-ba28-49edc3b02f43 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:32.632130 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdabc647-8c4c-462b-ba28-49edc3b02f43 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:32.647107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-fdabc647-8c4c-462b-ba28-49edc3b02f43 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:32.647107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:32.647107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:32.647107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:32.647107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:32.647107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:32.647107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:32.647107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:32.647107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:32.647107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:32.647107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:32.647107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:32.647107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:10:32.647107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:10:32.647107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:10:32.647107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:10:32.648139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:32.648139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:32.648139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:32.648139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:32.648139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:32.648139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:32.648139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:32.648139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:32.648139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:32.648139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:32.648139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:32.648139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:32.648139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:32.648139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:32.648139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:32.648139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:32.648139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:10:32.649382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:10:32.649382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:32.649382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:10:32.649382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 635/1265] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:10:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 04:10:32.653764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e54c31-d598-47f3-8cc1-6ff0320770e2 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:32.731413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:32.749829 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e54c31-d598-47f3-8cc1-6ff0320770e2 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:32.782071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648ebb7e-f43e-409c-9642-efde11c7cbca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:32.782650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648ebb7e-f43e-409c-9642-efde11c7cbca None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:32.782919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648ebb7e-f43e-409c-9642-efde11c7cbca None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:32.783185 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648ebb7e-f43e-409c-9642-efde11c7cbca None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:32.783806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648ebb7e-f43e-409c-9642-efde11c7cbca None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:32.784098 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648ebb7e-f43e-409c-9642-efde11c7cbca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:32.785433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648ebb7e-f43e-409c-9642-efde11c7cbca None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:32.796404 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 631/1266] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:10:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:32.801949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdabc647-8c4c-462b-ba28-49edc3b02f43 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:32.898322 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdabc647-8c4c-462b-ba28-49edc3b02f43 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:32.916826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:32.930493 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e4a736-147f-4600-8e73-e07569d913c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:32.931105 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e4a736-147f-4600-8e73-e07569d913c0 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:32.931385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e4a736-147f-4600-8e73-e07569d913c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:32.931654 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e4a736-147f-4600-8e73-e07569d913c0 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:32.938739 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e4a736-147f-4600-8e73-e07569d913c0 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:32.940149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e4a736-147f-4600-8e73-e07569d913c0 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:32.951436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-c4e4a736-147f-4600-8e73-e07569d913c0 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 27 04:10:32.951436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:32.951436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:32.951436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:32.951436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:32.951436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:32.951436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:32.951436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:32.951436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:32.951436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:32.951436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:32.951436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:32.951436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:10:32.951436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:10:32.951436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:10:32.951436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:10:32.952531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:32.952531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:32.952531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:10:32.952531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:10:32.952531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:10:32.952531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:10:32.952531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:32.952531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:32.952531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:32.952531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:32.952531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:32.952531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:32.952531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:32.952531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:32.952531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:32.952531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:32.952531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:32.952531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:32.953502 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:32.953502 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:32.953502 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:10:32.953502 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:10:32.953502 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:10:32.953502 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:10:32.953502 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:10:32.953502 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:10:32.953502 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 27 04:10:32.953502 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:10:32.953502 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 636/1267] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:10:32 2022] GET /identity/v3/users/placement => generated 87 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:32.958901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648ebb7e-f43e-409c-9642-efde11c7cbca None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:33.054107 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648ebb7e-f43e-409c-9642-efde11c7cbca None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:33.087034 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af5d4cf-47f6-4053-9ff7-0759c2be73b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:33.087632 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af5d4cf-47f6-4053-9ff7-0759c2be73b3 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:33.087941 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af5d4cf-47f6-4053-9ff7-0759c2be73b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:33.088213 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af5d4cf-47f6-4053-9ff7-0759c2be73b3 None admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:33.089041 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af5d4cf-47f6-4053-9ff7-0759c2be73b3 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:33.089391 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af5d4cf-47f6-4053-9ff7-0759c2be73b3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:33.090831 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af5d4cf-47f6-4053-9ff7-0759c2be73b3 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:33.101599 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:33.110841 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 632/1268] 10.222.0.26 () {56 vars in 1143 bytes} [Tue Sep 27 04:10:32 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:33.117070 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e4a736-147f-4600-8e73-e07569d913c0 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:33.212843 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e4a736-147f-4600-8e73-e07569d913c0 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:33.244635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071ac8f9-7f05-4a32-ac46-ebc2ccdc6ad0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:33.245231 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071ac8f9-7f05-4a32-ac46-ebc2ccdc6ad0 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:33.245515 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071ac8f9-7f05-4a32-ac46-ebc2ccdc6ad0 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:33.245813 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071ac8f9-7f05-4a32-ac46-ebc2ccdc6ad0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:33.256726 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071ac8f9-7f05-4a32-ac46-ebc2ccdc6ad0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:33.258162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071ac8f9-7f05-4a32-ac46-ebc2ccdc6ad0 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:33.271976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-071ac8f9-7f05-4a32-ac46-ebc2ccdc6ad0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:33.271976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:33.271976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:33.271976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:33.271976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:33.271976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:33.271976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:33.271976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:33.271976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:33.271976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:33.271976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:33.271976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:33.271976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:10:33.271976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:10:33.271976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:10:33.271976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:10:33.273073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:33.273073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:33.273073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:33.273073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:33.273073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:33.273073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:33.273073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:33.273073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:33.273073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:33.273073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:33.273073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:33.273073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:33.273073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:33.273073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:33.273073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:33.273073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:33.273073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:10:33.274357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:10:33.274357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:33.274357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:10:33.274357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 637/1269] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:10:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:33.278537 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af5d4cf-47f6-4053-9ff7-0759c2be73b3 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:33.375379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af5d4cf-47f6-4053-9ff7-0759c2be73b3 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:33.408053 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445942f1-3187-4697-b2f8-3f256a47a47f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:33.408648 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445942f1-3187-4697-b2f8-3f256a47a47f None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:33.408920 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445942f1-3187-4697-b2f8-3f256a47a47f None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:33.409187 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445942f1-3187-4697-b2f8-3f256a47a47f None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:33.409848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445942f1-3187-4697-b2f8-3f256a47a47f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:33.410146 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445942f1-3187-4697-b2f8-3f256a47a47f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:33.412007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445942f1-3187-4697-b2f8-3f256a47a47f None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:33.422862 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 633/1270] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:10:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:33.428978 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071ac8f9-7f05-4a32-ac46-ebc2ccdc6ad0 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:33.459439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:33.524956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071ac8f9-7f05-4a32-ac46-ebc2ccdc6ad0 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:33.557287 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb24b3f-9de1-4abf-80fb-cd2d99d7824c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:33.557919 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb24b3f-9de1-4abf-80fb-cd2d99d7824c None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:33.558198 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb24b3f-9de1-4abf-80fb-cd2d99d7824c None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:33.558464 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb24b3f-9de1-4abf-80fb-cd2d99d7824c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:33.564822 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb24b3f-9de1-4abf-80fb-cd2d99d7824c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:33.566269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb24b3f-9de1-4abf-80fb-cd2d99d7824c None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:33.577108 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-2fb24b3f-9de1-4abf-80fb-cd2d99d7824c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:10:33.577108 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:33.577108 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:33.577108 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:33.577108 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:33.577108 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:33.577108 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:33.577108 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:33.577108 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:33.577108 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:33.577108 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:33.577108 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:33.577108 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:10:33.577108 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:10:33.577108 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:10:33.578156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:10:33.578156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:33.578156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:33.578156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:33.578156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:33.578156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:33.578156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:33.578156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:33.578156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:33.578156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:33.578156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:33.578156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:33.578156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:33.578156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:33.578156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:33.578156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:33.578156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:33.579272 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:10:33.579272 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:10:33.579272 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:10:33.579272 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:10:33.579272 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:10:33.579272 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:10:33.579272 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:10:33.579272 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:10:33.579272 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 638/1271] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:10:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:33.583929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445942f1-3187-4697-b2f8-3f256a47a47f None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:33.639795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:33.679948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445942f1-3187-4697-b2f8-3f256a47a47f None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:33.711595 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b3418d7-24a1-40d9-8c3b-0e252a312158 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:33.712174 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3418d7-24a1-40d9-8c3b-0e252a312158 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:33.712442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3418d7-24a1-40d9-8c3b-0e252a312158 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:33.712706 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3418d7-24a1-40d9-8c3b-0e252a312158 None admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:33.713373 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b3418d7-24a1-40d9-8c3b-0e252a312158 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:33.713672 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b3418d7-24a1-40d9-8c3b-0e252a312158 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:33.715096 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b3418d7-24a1-40d9-8c3b-0e252a312158 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:33.726952 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 634/1272] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:10:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:33.732840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb24b3f-9de1-4abf-80fb-cd2d99d7824c None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:33.819375 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:33.827592 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb24b3f-9de1-4abf-80fb-cd2d99d7824c None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:33.859675 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdd8870-f635-4ef0-ace4-677e8635b2f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:33.860303 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdd8870-f635-4ef0-ace4-677e8635b2f3 None admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:33.860576 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdd8870-f635-4ef0-ace4-677e8635b2f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:33.860864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdd8870-f635-4ef0-ace4-677e8635b2f3 None admin] PATH_INFO: `/v3/projects/c830a3bfece6456486f4b094076b5364/users/3a1bc198dfc0486e9e56d4fb28cda05d/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:33.862707 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfdd8870-f635-4ef0-ace4-677e8635b2f3 None admin] RBAC: Authorizing `identity:create_grant(project_id=c830a3bfece6456486f4b094076b5364, user_id=3a1bc198dfc0486e9e56d4fb28cda05d, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:33.864126 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfdd8870-f635-4ef0-ace4-677e8635b2f3 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:33.875484 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 639/1273] 10.222.0.26 () {58 vars in 1439 bytes} [Tue Sep 27 04:10:33 2022] PUT /identity/v3/projects/c830a3bfece6456486f4b094076b5364/users/3a1bc198dfc0486e9e56d4fb28cda05d/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:10:36.027747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25193f03-3fc9-4c62-b389-c92aa2678116 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:36.028284 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25193f03-3fc9-4c62-b389-c92aa2678116 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:36.028284 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25193f03-3fc9-4c62-b389-c92aa2678116 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:36.029589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 635/1274] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:10:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:10:36.034787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6034c265-a6d1-4846-88e6-0c0a78a5bc44 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:36.035202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6034c265-a6d1-4846-88e6-0c0a78a5bc44 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:36.035335 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6034c265-a6d1-4846-88e6-0c0a78a5bc44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:36.053734 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-6034c265-a6d1-4846-88e6-0c0a78a5bc44 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:10:36.054939 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-6034c265-a6d1-4846-88e6-0c0a78a5bc44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:10:36.125586 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 640/1275] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:10:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:10:36.396641 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25193f03-3fc9-4c62-b389-c92aa2678116 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:36.494501 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25193f03-3fc9-4c62-b389-c92aa2678116 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:36.511838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:36.527999 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d875b0fa-5152-49e8-8bb5-b84b2b139594 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:36.528612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d875b0fa-5152-49e8-8bb5-b84b2b139594 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:36.528884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d875b0fa-5152-49e8-8bb5-b84b2b139594 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:36.529152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d875b0fa-5152-49e8-8bb5-b84b2b139594 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:36.539323 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d875b0fa-5152-49e8-8bb5-b84b2b139594 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 04:10:36.544064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-d875b0fa-5152-49e8-8bb5-b84b2b139594 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:10:36.544064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:36.544064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:36.544064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:36.544064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:36.544064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:36.544064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:36.544064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:36.544064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:36.544064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:36.544064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:36.544064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:36.544064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 04:10:36.544064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 04:10:36.544064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 04:10:36.544064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 04:10:36.545114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 04:10:36.545114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 04:10:36.545114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 04:10:36.545114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 04:10:36.545114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:36.545114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:36.545114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:36.545114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:36.545114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:36.545114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:36.545114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:36.545114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:36.545114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:36.545114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:36.545114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:36.545114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:36.546017 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:36.546017 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:36.546017 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:36.546017 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:36.546017 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 04:10:36.546017 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 04:10:36.546017 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 04:10:36.546017 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 04:10:36.546017 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 04:10:36.546017 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 04:10:36.546017 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 04:10:36.546017 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:10:36.546017 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 636/1276] 10.222.0.26 () {56 vars in 1096 bytes} [Tue Sep 27 04:10:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:36.551028 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6034c265-a6d1-4846-88e6-0c0a78a5bc44 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:36.628150 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:36.646307 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6034c265-a6d1-4846-88e6-0c0a78a5bc44 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:36.680381 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ce53e0-f854-4712-8159-6116c89b5ea9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:36.680905 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ce53e0-f854-4712-8159-6116c89b5ea9 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:36.681183 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ce53e0-f854-4712-8159-6116c89b5ea9 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:36.681449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ce53e0-f854-4712-8159-6116c89b5ea9 None admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:36.682130 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ce53e0-f854-4712-8159-6116c89b5ea9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:36.682431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ce53e0-f854-4712-8159-6116c89b5ea9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:36.683771 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ce53e0-f854-4712-8159-6116c89b5ea9 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:36.691497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 641/1277] 10.222.0.26 () {56 vars in 1099 bytes} [Tue Sep 27 04:10:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:36.697283 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d875b0fa-5152-49e8-8bb5-b84b2b139594 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:36.792654 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d875b0fa-5152-49e8-8bb5-b84b2b139594 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:36.824130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc157a6c-e02b-477d-ae7f-0fa28d5e55bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:36.824727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc157a6c-e02b-477d-ae7f-0fa28d5e55bc None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:36.824997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc157a6c-e02b-477d-ae7f-0fa28d5e55bc None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:36.825263 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc157a6c-e02b-477d-ae7f-0fa28d5e55bc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:36.835745 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc157a6c-e02b-477d-ae7f-0fa28d5e55bc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:36.837170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc157a6c-e02b-477d-ae7f-0fa28d5e55bc None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:36.851294 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-cc157a6c-e02b-477d-ae7f-0fa28d5e55bc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:36.851294 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:36.851294 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:36.851294 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:36.851294 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:36.851294 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:36.851294 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:36.851294 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:36.851294 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:36.851294 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:36.851294 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:36.851294 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:36.851294 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:10:36.851294 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:10:36.851294 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:10:36.851294 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:10:36.852452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:36.852452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:36.852452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:36.852452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:36.852452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:36.852452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:36.852452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:36.852452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:36.852452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:36.852452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:36.852452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:36.852452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:36.852452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:36.852452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:36.852452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:36.852452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:36.852452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:10:36.853348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:10:36.853348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:36.853348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:10:36.853348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 637/1278] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:10:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:36.857989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ce53e0-f854-4712-8159-6116c89b5ea9 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:36.954429 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ce53e0-f854-4712-8159-6116c89b5ea9 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:36.986974 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b635cbee-ed87-4a32-8d15-80bb53c0c3cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:36.987555 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b635cbee-ed87-4a32-8d15-80bb53c0c3cb None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:36.987830 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b635cbee-ed87-4a32-8d15-80bb53c0c3cb None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:36.988100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b635cbee-ed87-4a32-8d15-80bb53c0c3cb None admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:36.988728 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b635cbee-ed87-4a32-8d15-80bb53c0c3cb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:36.989024 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b635cbee-ed87-4a32-8d15-80bb53c0c3cb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:36.990438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b635cbee-ed87-4a32-8d15-80bb53c0c3cb None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:37.001703 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 642/1279] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:10:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:37.007398 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc157a6c-e02b-477d-ae7f-0fa28d5e55bc None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:37.037646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:37.103263 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc157a6c-e02b-477d-ae7f-0fa28d5e55bc None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:37.135831 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9b2b1a-d27b-49f9-bd7e-d0fee05c75cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:37.136441 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9b2b1a-d27b-49f9-bd7e-d0fee05c75cc None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:37.136714 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9b2b1a-d27b-49f9-bd7e-d0fee05c75cc None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:37.136983 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9b2b1a-d27b-49f9-bd7e-d0fee05c75cc None admin] PATH_INFO: `/v3/users/placement` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:37.144097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9b2b1a-d27b-49f9-bd7e-d0fee05c75cc None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:37.146122 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9b2b1a-d27b-49f9-bd7e-d0fee05c75cc None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:37.156853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-5f9b2b1a-d27b-49f9-bd7e-d0fee05c75cc None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 27 04:10:37.156853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:37.156853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:37.156853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:37.156853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:37.156853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:37.156853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:37.156853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:37.156853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:37.156853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:37.156853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:37.156853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:37.156853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 04:10:37.156853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 04:10:37.156853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 04:10:37.156853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 04:10:37.157991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:37.157991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:37.157991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 04:10:37.157991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:10:37.157991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 04:10:37.157991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 04:10:37.157991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:37.157991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:37.157991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:37.157991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:37.157991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:37.157991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:37.157991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:37.157991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:37.157991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:37.157991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:37.157991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:37.157991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:37.159184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:37.159184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:37.159184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 04:10:37.159184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 04:10:37.159184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 04:10:37.159184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 04:10:37.159184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 04:10:37.159184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 04:10:37.159184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 27 04:10:37.159184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:10:37.159184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 638/1280] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:10:37 2022] GET /identity/v3/users/placement => generated 87 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:37.163863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b635cbee-ed87-4a32-8d15-80bb53c0c3cb None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:37.167596 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:37.257412 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b635cbee-ed87-4a32-8d15-80bb53c0c3cb None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:37.289431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb67a05b-db51-4a20-ac79-795eb094d0d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:37.290034 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb67a05b-db51-4a20-ac79-795eb094d0d8 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:37.290310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb67a05b-db51-4a20-ac79-795eb094d0d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:37.290574 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb67a05b-db51-4a20-ac79-795eb094d0d8 None admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:37.291283 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb67a05b-db51-4a20-ac79-795eb094d0d8 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:37.291583 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb67a05b-db51-4a20-ac79-795eb094d0d8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:37.292970 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb67a05b-db51-4a20-ac79-795eb094d0d8 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:37.309041 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 643/1281] 10.222.0.26 () {56 vars in 1143 bytes} [Tue Sep 27 04:10:37 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:37.314744 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9b2b1a-d27b-49f9-bd7e-d0fee05c75cc None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:37.401815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:37.411804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9b2b1a-d27b-49f9-bd7e-d0fee05c75cc None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:37.444137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936b6268-a71c-45b0-a5df-4bac30a0d60e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:37.444726 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936b6268-a71c-45b0-a5df-4bac30a0d60e None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:37.445002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936b6268-a71c-45b0-a5df-4bac30a0d60e None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:37.445275 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936b6268-a71c-45b0-a5df-4bac30a0d60e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:37.509045 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936b6268-a71c-45b0-a5df-4bac30a0d60e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:37.510692 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936b6268-a71c-45b0-a5df-4bac30a0d60e None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:37.593326 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:37.603570 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 639/1282] 10.222.0.26 () {58 vars in 1282 bytes} [Tue Sep 27 04:10:37 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 27 04:10:37.609311 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb67a05b-db51-4a20-ac79-795eb094d0d8 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:37.671963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:37.709328 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb67a05b-db51-4a20-ac79-795eb094d0d8 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:37.742288 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81376f67-9ede-4efe-93b6-4358980c3255 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:37.742892 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81376f67-9ede-4efe-93b6-4358980c3255 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:37.743169 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81376f67-9ede-4efe-93b6-4358980c3255 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:37.743435 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81376f67-9ede-4efe-93b6-4358980c3255 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:37.752866 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81376f67-9ede-4efe-93b6-4358980c3255 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:37.754306 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81376f67-9ede-4efe-93b6-4358980c3255 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:37.769129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-81376f67-9ede-4efe-93b6-4358980c3255 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:37.769129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:37.769129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:37.769129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:37.769129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:37.769129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:37.769129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:37.769129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:37.769129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:37.769129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:37.769129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:37.769129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:37.769129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 04:10:37.769129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 04:10:37.769129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 04:10:37.769129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 04:10:37.770247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:37.770247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:37.770247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:37.770247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:37.770247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:37.770247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:37.770247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:37.770247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:37.770247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:37.770247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:37.770247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:37.770247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:37.770247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:37.770247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:37.770247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:37.770247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:37.770247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 04:10:37.771341 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 04:10:37.771341 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 04:10:37.771341 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:10:37.771341 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 644/1283] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:10:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:37.775916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936b6268-a71c-45b0-a5df-4bac30a0d60e None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:37.871972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936b6268-a71c-45b0-a5df-4bac30a0d60e None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:37.904079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace9ce18-8dd9-422b-9600-ed91e28bb6e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:37.904670 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace9ce18-8dd9-422b-9600-ed91e28bb6e9 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:37.904946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace9ce18-8dd9-422b-9600-ed91e28bb6e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:37.905219 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace9ce18-8dd9-422b-9600-ed91e28bb6e9 None admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:37.905897 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace9ce18-8dd9-422b-9600-ed91e28bb6e9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:37.906197 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace9ce18-8dd9-422b-9600-ed91e28bb6e9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:37.907547 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace9ce18-8dd9-422b-9600-ed91e28bb6e9 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:37.918858 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 640/1284] 10.222.0.26 () {56 vars in 1109 bytes} [Tue Sep 27 04:10:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:37.924324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81376f67-9ede-4efe-93b6-4358980c3255 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:38.020978 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81376f67-9ede-4efe-93b6-4358980c3255 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:38.033840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:38.054000 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbf5d2a-8f61-4d59-b9a7-9fbc810668fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:38.054611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbf5d2a-8f61-4d59-b9a7-9fbc810668fb None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:38.054885 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbf5d2a-8f61-4d59-b9a7-9fbc810668fb None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:38.055152 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbf5d2a-8f61-4d59-b9a7-9fbc810668fb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:38.061416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bbf5d2a-8f61-4d59-b9a7-9fbc810668fb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:38.062861 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bbf5d2a-8f61-4d59-b9a7-9fbc810668fb None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:38.073670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application [None req-2bbf5d2a-8f61-4d59-b9a7-9fbc810668fb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:10:38.073670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:38.073670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:38.073670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:38.073670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:38.073670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:38.073670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:38.073670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:38.073670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:38.073670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:38.073670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:38.073670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:38.073670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 04:10:38.073670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 04:10:38.073670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 04:10:38.074730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 04:10:38.074730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:38.074730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:38.074730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:38.074730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:38.074730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:38.074730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:38.074730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:38.074730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:38.074730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:38.074730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:38.074730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:38.074730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:38.074730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:38.074730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:38.074730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:38.074730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:38.075865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 04:10:38.075865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 04:10:38.075865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 04:10:38.075865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 04:10:38.075865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 04:10:38.075865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 04:10:38.075865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 04:10:38.075865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: ERROR keystone.server.flask.application Sep 27 04:10:38.075865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 645/1285] 10.222.0.26 () {56 vars in 1111 bytes} [Tue Sep 27 04:10:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:38.080601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace9ce18-8dd9-422b-9600-ed91e28bb6e9 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:38.132055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:38.180952 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace9ce18-8dd9-422b-9600-ed91e28bb6e9 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:38.213528 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72fc0e5-665a-4afa-aa6b-e236c9210fa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:38.214141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72fc0e5-665a-4afa-aa6b-e236c9210fa1 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:38.214419 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72fc0e5-665a-4afa-aa6b-e236c9210fa1 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:38.214691 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72fc0e5-665a-4afa-aa6b-e236c9210fa1 None admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:38.215372 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72fc0e5-665a-4afa-aa6b-e236c9210fa1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:38.215675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72fc0e5-665a-4afa-aa6b-e236c9210fa1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:38.217082 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72fc0e5-665a-4afa-aa6b-e236c9210fa1 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:38.228468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 641/1286] 10.222.0.26 () {56 vars in 1148 bytes} [Tue Sep 27 04:10:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:38.234496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbf5d2a-8f61-4d59-b9a7-9fbc810668fb None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:38.336745 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbf5d2a-8f61-4d59-b9a7-9fbc810668fb None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:38.371069 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26f87fd-60fc-49db-b969-48f23971cf82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:38.371682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26f87fd-60fc-49db-b969-48f23971cf82 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:38.371963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26f87fd-60fc-49db-b969-48f23971cf82 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:38.372238 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26f87fd-60fc-49db-b969-48f23971cf82 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:38.372959 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26f87fd-60fc-49db-b969-48f23971cf82 None admin] RBAC: Adding query filter params (role.id=e6a31562a18649f0809202c36a8e18ef, scope.project.id=c830a3bfece6456486f4b094076b5364, user.id=3a1bc198dfc0486e9e56d4fb28cda05d) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:38.373269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26f87fd-60fc-49db-b969-48f23971cf82 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:38.374787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26f87fd-60fc-49db-b969-48f23971cf82 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:38.381368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 646/1287] 10.222.0.26 () {56 vars in 1374 bytes} [Tue Sep 27 04:10:38 2022] GET /identity/v3/role_assignments?user.id=3a1bc198dfc0486e9e56d4fb28cda05d&scope.project.id=c830a3bfece6456486f4b094076b5364&role.id=e6a31562a18649f0809202c36a8e18ef => generated 608 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:40.529026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a885c53-7397-4c65-a9ff-c428a35cf75f None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:40.529548 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a885c53-7397-4c65-a9ff-c428a35cf75f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:40.529548 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a885c53-7397-4c65-a9ff-c428a35cf75f None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:40.530897 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 642/1288] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:10:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:10:40.536724 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0823f1-33a9-4b49-827b-313eaf2dc511 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:40.537160 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0823f1-33a9-4b49-827b-313eaf2dc511 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:40.537254 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0823f1-33a9-4b49-827b-313eaf2dc511 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:40.554808 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-af0823f1-33a9-4b49-827b-313eaf2dc511 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:10:40.555973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-af0823f1-33a9-4b49-827b-313eaf2dc511 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:10:40.584062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:40.620057 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 647/1289] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:10:40 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:10:40.886576 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a885c53-7397-4c65-a9ff-c428a35cf75f None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:40.982007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a885c53-7397-4c65-a9ff-c428a35cf75f None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:40.991991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:41.016466 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2aa81cb-0663-41a0-b5ac-512d569cd80f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:41.017072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2aa81cb-0663-41a0-b5ac-512d569cd80f None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:41.017343 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2aa81cb-0663-41a0-b5ac-512d569cd80f None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:41.017607 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2aa81cb-0663-41a0-b5ac-512d569cd80f None admin] PATH_INFO: `/v3/services/placement` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:41.018201 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2aa81cb-0663-41a0-b5ac-512d569cd80f None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:41.019537 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2aa81cb-0663-41a0-b5ac-512d569cd80f None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:41.028612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-a2aa81cb-0663-41a0-b5ac-512d569cd80f None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 27 04:10:41.028612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:41.028612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:10:41.028612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:10:41.028612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:10:41.028612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:10:41.028612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:10:41.028612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:10:41.028612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 27 04:10:41.028612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:10:41.028612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:10:41.028612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:10:41.028612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:41.028612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:41.028612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:41.028612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:41.028612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:41.029916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:41.029916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:41.029916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:41.029916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:41.029916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:41.029916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:41.029916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:10:41.029916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:10:41.029916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:10:41.029916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:10:41.029916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:41.029916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:41.029916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:41.029916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:41.029916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:41.029916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:41.030992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:41.030992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:41.030992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:41.030992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:41.030992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:41.030992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:41.030992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:41.030992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:41.030992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:41.030992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:41.030992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:10:41.030992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:10:41.030992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 27 04:10:41.030992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:10:41.030992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 643/1290] 10.222.0.26 () {56 vars in 1117 bytes} [Tue Sep 27 04:10:40 2022] GET /identity/v3/services/placement => generated 90 bytes in 144 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:41.035538 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0823f1-33a9-4b49-827b-313eaf2dc511 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:41.133736 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0823f1-33a9-4b49-827b-313eaf2dc511 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:41.167452 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3b6dd6-27e4-40e2-8cb8-bf061d90d9cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:41.167982 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3b6dd6-27e4-40e2-8cb8-bf061d90d9cf None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:41.168261 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3b6dd6-27e4-40e2-8cb8-bf061d90d9cf None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:41.168533 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3b6dd6-27e4-40e2-8cb8-bf061d90d9cf None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:41.169155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3b6dd6-27e4-40e2-8cb8-bf061d90d9cf None admin] RBAC: Adding query filter params (name=placement) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:41.169458 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3b6dd6-27e4-40e2-8cb8-bf061d90d9cf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:41.170836 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3b6dd6-27e4-40e2-8cb8-bf061d90d9cf None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:41.175446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 648/1291] 10.222.0.26 () {56 vars in 1116 bytes} [Tue Sep 27 04:10:41 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:41.181219 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2aa81cb-0663-41a0-b5ac-512d569cd80f None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:41.277485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2aa81cb-0663-41a0-b5ac-512d569cd80f None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:41.309335 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff5d97f-2ce8-4a72-a669-9af8ba0973a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:41.309945 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff5d97f-2ce8-4a72-a669-9af8ba0973a7 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:41.310221 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff5d97f-2ce8-4a72-a669-9af8ba0973a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:41.310491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff5d97f-2ce8-4a72-a669-9af8ba0973a7 None admin] PATH_INFO: `/v3/services` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:41.311102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff5d97f-2ce8-4a72-a669-9af8ba0973a7 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:41.311398 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff5d97f-2ce8-4a72-a669-9af8ba0973a7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:41.312740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff5d97f-2ce8-4a72-a669-9af8ba0973a7 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:41.316940 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 644/1292] 10.222.0.26 () {56 vars in 1116 bytes} [Tue Sep 27 04:10:41 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:43.413470 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676b7cdd-01be-4f96-ae0e-14657fafb79c None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:43.413915 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676b7cdd-01be-4f96-ae0e-14657fafb79c None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:43.414043 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676b7cdd-01be-4f96-ae0e-14657fafb79c None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:43.415353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 649/1293] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:10:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:10:43.420541 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5302c8-7e4e-4992-b2b2-57cf62203ae2 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:43.420943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5302c8-7e4e-4992-b2b2-57cf62203ae2 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:43.421077 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5302c8-7e4e-4992-b2b2-57cf62203ae2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:43.438639 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-3e5302c8-7e4e-4992-b2b2-57cf62203ae2 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:10:43.439786 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-3e5302c8-7e4e-4992-b2b2-57cf62203ae2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:10:43.468447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:43.504521 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 645/1294] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:10:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:10:43.770725 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676b7cdd-01be-4f96-ae0e-14657fafb79c None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:43.800486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:43.869233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676b7cdd-01be-4f96-ae0e-14657fafb79c None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:43.901921 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b0f2c4-e7df-45d1-b368-d152d5107c04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:43.902508 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b0f2c4-e7df-45d1-b368-d152d5107c04 None admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:43.902785 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b0f2c4-e7df-45d1-b368-d152d5107c04 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:43.903055 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b0f2c4-e7df-45d1-b368-d152d5107c04 None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:43.903699 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b0f2c4-e7df-45d1-b368-d152d5107c04 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:43.905038 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b0f2c4-e7df-45d1-b368-d152d5107c04 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:43.919073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 650/1295] 10.222.0.26 () {60 vars in 1141 bytes} [Tue Sep 27 04:10:43 2022] POST /identity/v3/services => generated 246 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:10:46.026819 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaacd048-0220-48e4-8b29-d29a74b2b7b4 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:46.027351 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaacd048-0220-48e4-8b29-d29a74b2b7b4 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:46.027351 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaacd048-0220-48e4-8b29-d29a74b2b7b4 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:46.028675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 646/1296] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:10:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:10:46.033949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82218951-2523-4b04-a5ec-f8f2147486c9 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:46.034333 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82218951-2523-4b04-a5ec-f8f2147486c9 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:46.034497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82218951-2523-4b04-a5ec-f8f2147486c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:46.052523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-82218951-2523-4b04-a5ec-f8f2147486c9 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:10:46.053690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-82218951-2523-4b04-a5ec-f8f2147486c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:10:46.128889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 651/1297] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:10:46 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:10:46.400700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaacd048-0220-48e4-8b29-d29a74b2b7b4 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:46.497462 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaacd048-0220-48e4-8b29-d29a74b2b7b4 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:46.530286 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813ff6b8-3a38-4e6e-95b4-87b77340d40b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:46.530897 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813ff6b8-3a38-4e6e-95b4-87b77340d40b None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:46.531171 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813ff6b8-3a38-4e6e-95b4-87b77340d40b None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:46.531438 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813ff6b8-3a38-4e6e-95b4-87b77340d40b None admin] PATH_INFO: `/v3/services/placement` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:46.532003 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813ff6b8-3a38-4e6e-95b4-87b77340d40b None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:46.533344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813ff6b8-3a38-4e6e-95b4-87b77340d40b None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:46.542442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-813ff6b8-3a38-4e6e-95b4-87b77340d40b None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 27 04:10:46.542442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:46.542442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:10:46.542442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:10:46.542442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:10:46.542442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:10:46.542442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:10:46.542442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:10:46.542442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 27 04:10:46.542442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:10:46.542442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:10:46.542442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:10:46.542442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:46.542442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:46.542442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:46.542442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:46.542442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:46.543615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:46.543615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:46.543615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:46.543615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:46.543615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:46.543615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:46.543615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:10:46.543615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:10:46.543615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:10:46.543615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:10:46.543615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:46.543615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:46.543615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:46.543615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:46.543615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:46.543615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:46.544675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:46.544675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:46.544675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:46.544675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:46.544675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:46.544675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:46.544675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:46.544675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:46.544675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:46.544675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:46.544675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:10:46.544675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:10:46.544675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 27 04:10:46.544675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:10:46.544675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 647/1298] 10.222.0.26 () {56 vars in 1117 bytes} [Tue Sep 27 04:10:46 2022] GET /identity/v3/services/placement => generated 90 bytes in 144 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:46.549598 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82218951-2523-4b04-a5ec-f8f2147486c9 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:46.567071 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:46.643589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82218951-2523-4b04-a5ec-f8f2147486c9 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:46.676381 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3aa5fba-7a65-4507-aeb4-81d05fa81907 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:46.676928 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3aa5fba-7a65-4507-aeb4-81d05fa81907 None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:46.677202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3aa5fba-7a65-4507-aeb4-81d05fa81907 None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:46.677468 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3aa5fba-7a65-4507-aeb4-81d05fa81907 None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:46.678112 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3aa5fba-7a65-4507-aeb4-81d05fa81907 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:46.678405 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3aa5fba-7a65-4507-aeb4-81d05fa81907 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:46.679733 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3aa5fba-7a65-4507-aeb4-81d05fa81907 None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:46.684577 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 652/1299] 10.222.0.26 () {56 vars in 1116 bytes} [Tue Sep 27 04:10:46 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:46.690350 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813ff6b8-3a38-4e6e-95b4-87b77340d40b None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:46.720230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:46.784801 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813ff6b8-3a38-4e6e-95b4-87b77340d40b None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:46.816396 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca699aa-2424-4aa1-812b-14b222b5f0ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:46.816973 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca699aa-2424-4aa1-812b-14b222b5f0ae None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:46.817830 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca699aa-2424-4aa1-812b-14b222b5f0ae None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:46.818104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca699aa-2424-4aa1-812b-14b222b5f0ae None admin] PATH_INFO: `/v3/endpoints` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:46.818786 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca699aa-2424-4aa1-812b-14b222b5f0ae None admin] RBAC: Adding query filter params (interface=public, service_id=c595a1e37c1341ada6e545f5df322a42, region_id=RegionOne) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:46.819085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca699aa-2424-4aa1-812b-14b222b5f0ae None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:46.820414 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca699aa-2424-4aa1-812b-14b222b5f0ae None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:46.825171 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 648/1300] 10.222.0.26 () {56 vars in 1251 bytes} [Tue Sep 27 04:10:46 2022] GET /identity/v3/endpoints?service_id=c595a1e37c1341ada6e545f5df322a42&interface=public®ion_id=RegionOne => generated 194 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:46.830897 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3aa5fba-7a65-4507-aeb4-81d05fa81907 None admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:46.908934 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:46.926303 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3aa5fba-7a65-4507-aeb4-81d05fa81907 None admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:46.959488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf36890f-5c9c-4966-a7a3-3f68826dbeed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:46.960008 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf36890f-5c9c-4966-a7a3-3f68826dbeed None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:46.960282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf36890f-5c9c-4966-a7a3-3f68826dbeed None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:46.960548 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf36890f-5c9c-4966-a7a3-3f68826dbeed None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:46.961140 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf36890f-5c9c-4966-a7a3-3f68826dbeed None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:46.962501 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf36890f-5c9c-4966-a7a3-3f68826dbeed None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:46.968959 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 653/1301] 10.222.0.26 () {56 vars in 1087 bytes} [Tue Sep 27 04:10:46 2022] GET /identity/v3/services => generated 2191 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:10:49.085002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed31a4cf-0b1b-45e1-a47d-6c17c3bb0a01 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:49.085539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed31a4cf-0b1b-45e1-a47d-6c17c3bb0a01 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:49.085539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed31a4cf-0b1b-45e1-a47d-6c17c3bb0a01 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:49.087037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 649/1302] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:10:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:10:49.093177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d00821-0cf2-43df-9672-347ebd0a4314 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:49.093568 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d00821-0cf2-43df-9672-347ebd0a4314 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:49.093760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d00821-0cf2-43df-9672-347ebd0a4314 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:49.116148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-89d00821-0cf2-43df-9672-347ebd0a4314 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:10:49.117432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-89d00821-0cf2-43df-9672-347ebd0a4314 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:10:49.184550 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 654/1303] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:10:49 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 04:10:49.447321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed31a4cf-0b1b-45e1-a47d-6c17c3bb0a01 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:49.539216 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:49.545933 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed31a4cf-0b1b-45e1-a47d-6c17c3bb0a01 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:49.578430 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841eb3ec-fbad-46a6-a0e0-2efa8d9aad73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:49.579037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841eb3ec-fbad-46a6-a0e0-2efa8d9aad73 None admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:49.579312 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841eb3ec-fbad-46a6-a0e0-2efa8d9aad73 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:49.579583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841eb3ec-fbad-46a6-a0e0-2efa8d9aad73 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:49.580148 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841eb3ec-fbad-46a6-a0e0-2efa8d9aad73 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:49.581485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841eb3ec-fbad-46a6-a0e0-2efa8d9aad73 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:49.590079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application [None req-841eb3ec-fbad-46a6-a0e0-2efa8d9aad73 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 27 04:10:49.590079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:49.590079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 04:10:49.590079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 04:10:49.590079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 04:10:49.590079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 04:10:49.590079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 04:10:49.590079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:10:49.590079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 27 04:10:49.590079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:10:49.590079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 04:10:49.590079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:10:49.590079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 04:10:49.590079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 04:10:49.590079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 04:10:49.590079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 04:10:49.590079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 04:10:49.591216 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 04:10:49.591216 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 04:10:49.591216 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 04:10:49.591216 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 04:10:49.591216 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 04:10:49.591216 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 04:10:49.591216 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 04:10:49.591216 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 04:10:49.591216 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 04:10:49.591216 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 04:10:49.591216 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 04:10:49.591216 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 04:10:49.591216 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 04:10:49.591216 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 04:10:49.591216 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 04:10:49.591216 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 04:10:49.592386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 04:10:49.592386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application with Lock( Sep 27 04:10:49.592386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 04:10:49.592386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self._enter() Sep 27 04:10:49.592386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 04:10:49.592386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 04:10:49.592386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 04:10:49.592386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application return self.creator() Sep 27 04:10:49.592386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 04:10:49.592386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application created_value = creator( Sep 27 04:10:49.592386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 04:10:49.592386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 04:10:49.592386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 27 04:10:49.592386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: ERROR keystone.server.flask.application Sep 27 04:10:49.592386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 650/1304] 10.222.0.26 () {56 vars in 1117 bytes} [Tue Sep 27 04:10:49 2022] GET /identity/v3/services/placement => generated 90 bytes in 144 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 04:10:49.596930 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d00821-0cf2-43df-9672-347ebd0a4314 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:49.672760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:49.696017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d00821-0cf2-43df-9672-347ebd0a4314 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:49.727940 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04cb028-a1ee-4284-8817-7a2ef58508ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:49.728533 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04cb028-a1ee-4284-8817-7a2ef58508ad None admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:49.728813 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04cb028-a1ee-4284-8817-7a2ef58508ad None admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:49.729086 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04cb028-a1ee-4284-8817-7a2ef58508ad None admin] PATH_INFO: `/v3/services` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:49.729714 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04cb028-a1ee-4284-8817-7a2ef58508ad None admin] RBAC: Adding query filter params (name=placement) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:10:49.730045 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04cb028-a1ee-4284-8817-7a2ef58508ad None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:49.731387 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04cb028-a1ee-4284-8817-7a2ef58508ad None admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:49.736310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 655/1305] 10.222.0.26 () {56 vars in 1116 bytes} [Tue Sep 27 04:10:49 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:10:49.742645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841eb3ec-fbad-46a6-a0e0-2efa8d9aad73 None admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:10:49.836647 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841eb3ec-fbad-46a6-a0e0-2efa8d9aad73 None admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:10:49.868863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0020ee35-4c1a-454b-9f64-cb75c1a70e99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:10:49.869439 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0020ee35-4c1a-454b-9f64-cb75c1a70e99 None admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:10:49.869709 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0020ee35-4c1a-454b-9f64-cb75c1a70e99 None admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:10:49.870004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0020ee35-4c1a-454b-9f64-cb75c1a70e99 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:10:49.870655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0020ee35-4c1a-454b-9f64-cb75c1a70e99 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:10:49.871988 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0020ee35-4c1a-454b-9f64-cb75c1a70e99 None admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:10:49.890395 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 651/1306] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:10:49 2022] POST /identity/v3/endpoints => generated 331 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:10:59.733780 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=30 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:10:59.886286 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:11:53.454232 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26220dfe-36a2-400d-bd68-9d2d39c3cc3c None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:11:53.454711 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26220dfe-36a2-400d-bd68-9d2d39c3cc3c None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:11:53.454819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26220dfe-36a2-400d-bd68-9d2d39c3cc3c None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:11:53.461353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 656/1307] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:11:53 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:11:53.462581 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0038548b-02cd-4c0d-94e3-32943f8c6405 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:11:53.463045 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0038548b-02cd-4c0d-94e3-32943f8c6405 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:11:53.463139 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0038548b-02cd-4c0d-94e3-32943f8c6405 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:11:53.582946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-0038548b-02cd-4c0d-94e3-32943f8c6405 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:11:53.585568 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-0038548b-02cd-4c0d-94e3-32943f8c6405 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:11:53.727209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 652/1308] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:11:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 266 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:11:54.025730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef36ed3a-6a05-4730-9a08-1595dbcd73d8 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:11:54.026150 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef36ed3a-6a05-4730-9a08-1595dbcd73d8 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:11:54.026549 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef36ed3a-6a05-4730-9a08-1595dbcd73d8 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:11:54.028167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 657/1309] 10.222.0.26 () {54 vars in 964 bytes} [Tue Sep 27 04:11:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:11:54.034406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f52561-c5c9-445b-96bf-13fa2f80552d None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:11:54.034702 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f52561-c5c9-445b-96bf-13fa2f80552d None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:11:54.034961 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f52561-c5c9-445b-96bf-13fa2f80552d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:11:54.114518 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-97f52561-c5c9-445b-96bf-13fa2f80552d None None] MFA Rules not processed for user `cf29a8da103b411fbc79004eb371c05d`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:11:54.116277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-97f52561-c5c9-445b-96bf-13fa2f80552d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:11:54.138357 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:11:54.196714 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 653/1310] 10.222.0.26 () {58 vars in 1061 bytes} [Tue Sep 27 04:11:54 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:11:54.205649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef36ed3a-6a05-4730-9a08-1595dbcd73d8 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:11:54.207488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ef36ed3a-6a05-4730-9a08-1595dbcd73d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:11:54.322416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef36ed3a-6a05-4730-9a08-1595dbcd73d8 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:11:54.400549 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60dcd78-1cba-4e70-abc7-294c85fee8cd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:11:54.401159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60dcd78-1cba-4e70-abc7-294c85fee8cd service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:11:54.401438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60dcd78-1cba-4e70-abc7-294c85fee8cd service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:11:54.401710 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60dcd78-1cba-4e70-abc7-294c85fee8cd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:11:54.403329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-f60dcd78-1cba-4e70-abc7-294c85fee8cd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:11:54.461545 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:11:54.518630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60dcd78-1cba-4e70-abc7-294c85fee8cd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:11:54.520049 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60dcd78-1cba-4e70-abc7-294c85fee8cd service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:11:54.539367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 658/1311] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:11:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:11:57.027635 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be23f42a-4d8d-47db-8507-3b94e9625015 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:11:57.028217 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be23f42a-4d8d-47db-8507-3b94e9625015 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:11:57.028217 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be23f42a-4d8d-47db-8507-3b94e9625015 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:11:57.029550 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 654/1312] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:11:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:11:57.035175 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912cce54-3a59-4ebf-bf7d-32cede52b9f2 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:11:57.035613 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912cce54-3a59-4ebf-bf7d-32cede52b9f2 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:11:57.035801 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912cce54-3a59-4ebf-bf7d-32cede52b9f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:11:57.120543 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-912cce54-3a59-4ebf-bf7d-32cede52b9f2 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:11:57.121768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-912cce54-3a59-4ebf-bf7d-32cede52b9f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:11:57.227967 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 659/1313] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:11:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:11:57.510780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be23f42a-4d8d-47db-8507-3b94e9625015 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:11:57.512606 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-be23f42a-4d8d-47db-8507-3b94e9625015 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:11:57.613637 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be23f42a-4d8d-47db-8507-3b94e9625015 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:11:57.683828 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:11:57.686040 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8fcc0cf-53f1-4905-9ffd-0e605184e2e1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:11:57.686858 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fcc0cf-53f1-4905-9ffd-0e605184e2e1 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:11:57.687324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fcc0cf-53f1-4905-9ffd-0e605184e2e1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:11:57.687650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fcc0cf-53f1-4905-9ffd-0e605184e2e1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:11:57.689395 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-b8fcc0cf-53f1-4905-9ffd-0e605184e2e1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:11:57.795006 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8fcc0cf-53f1-4905-9ffd-0e605184e2e1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:11:57.796426 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8fcc0cf-53f1-4905-9ffd-0e605184e2e1 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:11:57.813960 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 655/1314] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:11:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:12:00.268898 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0327e29-0601-406e-b562-264e397338e0 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:00.269472 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0327e29-0601-406e-b562-264e397338e0 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:00.269472 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0327e29-0601-406e-b562-264e397338e0 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:00.270813 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 660/1315] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:12:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:12:00.275984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6a56e9-470c-47be-9a2d-a29e6d572a26 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:00.276363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6a56e9-470c-47be-9a2d-a29e6d572a26 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:00.276553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6a56e9-470c-47be-9a2d-a29e6d572a26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:00.339081 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-bf6a56e9-470c-47be-9a2d-a29e6d572a26 None None] MFA Rules not processed for user `fe29a891245a498fa7af105f29387508`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:12:00.340253 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-bf6a56e9-470c-47be-9a2d-a29e6d572a26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:00.390155 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:00.434792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 656/1316] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:12:00 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:12:00.720997 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0327e29-0601-406e-b562-264e397338e0 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:12:00.722603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-a0327e29-0601-406e-b562-264e397338e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:00.734814 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:00.825303 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0327e29-0601-406e-b562-264e397338e0 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:12:00.896304 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76116be1-ac31-43b3-946b-fb15e4b9b236 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:12:00.897408 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76116be1-ac31-43b3-946b-fb15e4b9b236 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:00.897682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76116be1-ac31-43b3-946b-fb15e4b9b236 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:00.897977 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76116be1-ac31-43b3-946b-fb15e4b9b236 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:00.899548 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-76116be1-ac31-43b3-946b-fb15e4b9b236 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:00.958899 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:01.004966 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76116be1-ac31-43b3-946b-fb15e4b9b236 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:12:01.006430 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76116be1-ac31-43b3-946b-fb15e4b9b236 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:12:01.024338 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 661/1317] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:12:00 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:12:03.690967 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f18b87-652d-4036-9cac-e1ca8e3dfd73 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:03.691494 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f18b87-652d-4036-9cac-e1ca8e3dfd73 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:03.691494 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f18b87-652d-4036-9cac-e1ca8e3dfd73 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:03.692741 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 657/1318] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:12:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:12:03.698194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d8554a-2a57-452b-a427-8d48afcf3ea1 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:03.698611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d8554a-2a57-452b-a427-8d48afcf3ea1 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:03.698751 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d8554a-2a57-452b-a427-8d48afcf3ea1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:03.763971 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-c3d8554a-2a57-452b-a427-8d48afcf3ea1 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:12:03.765191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c3d8554a-2a57-452b-a427-8d48afcf3ea1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:03.860059 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 662/1319] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:12:03 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:12:04.151938 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f18b87-652d-4036-9cac-e1ca8e3dfd73 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:12:04.153452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-f1f18b87-652d-4036-9cac-e1ca8e3dfd73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:04.252643 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f18b87-652d-4036-9cac-e1ca8e3dfd73 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:12:04.324963 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f15213-e090-45bd-880a-1ad782c8ba28 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:12:04.325561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f15213-e090-45bd-880a-1ad782c8ba28 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:04.325868 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f15213-e090-45bd-880a-1ad782c8ba28 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:04.326143 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f15213-e090-45bd-880a-1ad782c8ba28 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:04.327713 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-26f15213-e090-45bd-880a-1ad782c8ba28 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:04.335006 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:04.436001 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f15213-e090-45bd-880a-1ad782c8ba28 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:12:04.437418 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f15213-e090-45bd-880a-1ad782c8ba28 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:12:04.455265 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 658/1320] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:12:04 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:12:07.237364 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc1b468-7d69-40a3-984b-ec1145e7688c None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:07.237945 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc1b468-7d69-40a3-984b-ec1145e7688c None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:07.237945 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc1b468-7d69-40a3-984b-ec1145e7688c None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:07.239251 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 663/1321] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:12:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:12:07.244870 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249dddba-fef6-4062-869e-6bf15f05cb14 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:07.245297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249dddba-fef6-4062-869e-6bf15f05cb14 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:07.245430 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249dddba-fef6-4062-869e-6bf15f05cb14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:07.315939 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-249dddba-fef6-4062-869e-6bf15f05cb14 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:12:07.317155 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-249dddba-fef6-4062-869e-6bf15f05cb14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:07.378477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:07.421815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 659/1322] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:12:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:12:07.700133 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc1b468-7d69-40a3-984b-ec1145e7688c None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:12:07.701762 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ddc1b468-7d69-40a3-984b-ec1145e7688c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:07.722637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:07.806181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc1b468-7d69-40a3-984b-ec1145e7688c None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:12:07.877782 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fc8ca3-8c78-4cf8-9090-9bce6053b895 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:12:07.878680 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fc8ca3-8c78-4cf8-9090-9bce6053b895 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:07.878954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fc8ca3-8c78-4cf8-9090-9bce6053b895 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:07.879219 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fc8ca3-8c78-4cf8-9090-9bce6053b895 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:07.880784 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-76fc8ca3-8c78-4cf8-9090-9bce6053b895 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:07.950977 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:07.988259 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76fc8ca3-8c78-4cf8-9090-9bce6053b895 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:12:07.989687 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76fc8ca3-8c78-4cf8-9090-9bce6053b895 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:12:08.008042 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 664/1323] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:12:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:12:10.220692 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca7be44-cbb8-4d09-9685-00e588c72026 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:10.221225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca7be44-cbb8-4d09-9685-00e588c72026 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:10.221225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca7be44-cbb8-4d09-9685-00e588c72026 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:10.222611 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 660/1324] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:12:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:12:10.228456 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7663ba-3895-463c-8958-56b16426d74f None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:10.228908 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7663ba-3895-463c-8958-56b16426d74f None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:10.229315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7663ba-3895-463c-8958-56b16426d74f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:10.294230 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-1c7663ba-3895-463c-8958-56b16426d74f None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:12:10.295423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-1c7663ba-3895-463c-8958-56b16426d74f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:10.404745 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 665/1325] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:12:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:12:10.700797 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c02ea47-d069-4b3b-8e54-09cad1f836af None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:10.701086 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c02ea47-d069-4b3b-8e54-09cad1f836af None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:10.701362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c02ea47-d069-4b3b-8e54-09cad1f836af None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:10.702729 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 661/1326] 10.222.0.26 () {54 vars in 964 bytes} [Tue Sep 27 04:12:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:12:10.708570 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4004e86-4a90-46a4-bd1b-7e0814500ba8 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:10.708854 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4004e86-4a90-46a4-bd1b-7e0814500ba8 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:10.709131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4004e86-4a90-46a4-bd1b-7e0814500ba8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:10.741996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:10.795255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-c4004e86-4a90-46a4-bd1b-7e0814500ba8 None None] MFA Rules not processed for user `cf29a8da103b411fbc79004eb371c05d`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:12:10.796478 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c4004e86-4a90-46a4-bd1b-7e0814500ba8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:10.871115 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 666/1327] 10.222.0.26 () {58 vars in 1061 bytes} [Tue Sep 27 04:12:10 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:12:10.879989 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c02ea47-d069-4b3b-8e54-09cad1f836af None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:12:10.882147 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-0c02ea47-d069-4b3b-8e54-09cad1f836af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:10.982669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c02ea47-d069-4b3b-8e54-09cad1f836af None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:12:11.054393 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69b9571-2402-451c-ba20-327c016f060d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:12:11.054989 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69b9571-2402-451c-ba20-327c016f060d service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:11.055261 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69b9571-2402-451c-ba20-327c016f060d service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:11.055543 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69b9571-2402-451c-ba20-327c016f060d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:11.057074 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-e69b9571-2402-451c-ba20-327c016f060d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:11.073832 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:11.175408 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69b9571-2402-451c-ba20-327c016f060d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:12:11.176837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69b9571-2402-451c-ba20-327c016f060d service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:12:11.195450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 662/1328] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:12:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:12:13.642140 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5cdc6d-6b25-431c-b891-5d95fd82a471 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:13.642600 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5cdc6d-6b25-431c-b891-5d95fd82a471 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:13.642699 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5cdc6d-6b25-431c-b891-5d95fd82a471 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:13.644299 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 667/1329] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:12:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:12:13.650402 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0526baed-f8a4-4fb9-ac5b-b15bb301f90e None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:13.650908 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0526baed-f8a4-4fb9-ac5b-b15bb301f90e None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:13.651004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0526baed-f8a4-4fb9-ac5b-b15bb301f90e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:13.720989 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-0526baed-f8a4-4fb9-ac5b-b15bb301f90e None None] MFA Rules not processed for user `fe29a891245a498fa7af105f29387508`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:12:13.722223 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-0526baed-f8a4-4fb9-ac5b-b15bb301f90e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:13.789283 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:13.823546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 663/1330] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:12:13 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:12:14.106660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5cdc6d-6b25-431c-b891-5d95fd82a471 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:12:14.108158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-fe5cdc6d-6b25-431c-b891-5d95fd82a471 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:14.211021 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5cdc6d-6b25-431c-b891-5d95fd82a471 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:12:14.231034 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:14.284651 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad46c010-bcf2-4108-896a-3e9faf531b04 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:12:14.285255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad46c010-bcf2-4108-896a-3e9faf531b04 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:14.285530 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad46c010-bcf2-4108-896a-3e9faf531b04 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:14.285822 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad46c010-bcf2-4108-896a-3e9faf531b04 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:14.287406 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ad46c010-bcf2-4108-896a-3e9faf531b04 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:14.398494 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad46c010-bcf2-4108-896a-3e9faf531b04 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:12:14.399832 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad46c010-bcf2-4108-896a-3e9faf531b04 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:12:14.421060 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 668/1331] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:12:14 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:12:16.967103 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5b1aef-1a08-49c4-a22b-680b7877c969 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:16.967618 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5b1aef-1a08-49c4-a22b-680b7877c969 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:16.967618 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5b1aef-1a08-49c4-a22b-680b7877c969 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:16.968893 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 664/1332] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:12:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:12:16.974042 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e976620-7891-4dd4-8b05-841d13df27e0 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:16.974434 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e976620-7891-4dd4-8b05-841d13df27e0 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:16.974585 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e976620-7891-4dd4-8b05-841d13df27e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:17.036173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:17.039435 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-6e976620-7891-4dd4-8b05-841d13df27e0 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:12:17.040689 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-6e976620-7891-4dd4-8b05-841d13df27e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:17.135732 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 669/1333] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:12:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:12:17.417401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed5b1aef-1a08-49c4-a22b-680b7877c969 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:12:17.418964 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ed5b1aef-1a08-49c4-a22b-680b7877c969 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:17.518104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed5b1aef-1a08-49c4-a22b-680b7877c969 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:12:17.588737 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87bb3f9c-a73d-4cfa-84a3-6aa4f54f69ca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:12:17.589336 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bb3f9c-a73d-4cfa-84a3-6aa4f54f69ca service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:17.589614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bb3f9c-a73d-4cfa-84a3-6aa4f54f69ca service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:17.589917 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bb3f9c-a73d-4cfa-84a3-6aa4f54f69ca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:17.591535 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-87bb3f9c-a73d-4cfa-84a3-6aa4f54f69ca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:17.618498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:17.697614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87bb3f9c-a73d-4cfa-84a3-6aa4f54f69ca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:12:17.698994 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87bb3f9c-a73d-4cfa-84a3-6aa4f54f69ca service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:12:17.716336 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 665/1334] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:12:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:12:19.891365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414d2b60-8f70-4f7c-9346-9987b521fbdb None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:19.891903 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414d2b60-8f70-4f7c-9346-9987b521fbdb None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:19.892027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414d2b60-8f70-4f7c-9346-9987b521fbdb None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:19.893275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 670/1335] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:12:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:12:19.898964 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c93141-4f37-44df-b690-e788916c6b03 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:19.899391 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c93141-4f37-44df-b690-e788916c6b03 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:19.899509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c93141-4f37-44df-b690-e788916c6b03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:19.963965 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-62c93141-4f37-44df-b690-e788916c6b03 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:12:19.965135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-62c93141-4f37-44df-b690-e788916c6b03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:20.036857 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:20.058751 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 666/1336] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:12:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:12:20.343807 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414d2b60-8f70-4f7c-9346-9987b521fbdb None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:12:20.345387 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-414d2b60-8f70-4f7c-9346-9987b521fbdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:20.447194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414d2b60-8f70-4f7c-9346-9987b521fbdb None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:12:20.474647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:20.519728 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bfdd2f-6fa0-43bd-afe8-dd36e1182076 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:12:20.520265 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bfdd2f-6fa0-43bd-afe8-dd36e1182076 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:20.520542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bfdd2f-6fa0-43bd-afe8-dd36e1182076 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:20.520814 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bfdd2f-6fa0-43bd-afe8-dd36e1182076 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:20.522423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-b7bfdd2f-6fa0-43bd-afe8-dd36e1182076 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:20.628413 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7bfdd2f-6fa0-43bd-afe8-dd36e1182076 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:12:20.629858 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7bfdd2f-6fa0-43bd-afe8-dd36e1182076 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:12:20.647501 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 671/1337] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:12:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:12:23.125571 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede3600a-fcee-48c9-82cd-4922b1b0e35c None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:23.126171 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede3600a-fcee-48c9-82cd-4922b1b0e35c None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:23.126171 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede3600a-fcee-48c9-82cd-4922b1b0e35c None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:23.127505 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 667/1338] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:12:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:12:23.133362 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd43fb6e-f4cb-418f-bb4e-4f6e5757610b None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:23.133812 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd43fb6e-f4cb-418f-bb4e-4f6e5757610b None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:23.133952 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd43fb6e-f4cb-418f-bb4e-4f6e5757610b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:23.198851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-bd43fb6e-f4cb-418f-bb4e-4f6e5757610b None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:12:23.200106 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-bd43fb6e-f4cb-418f-bb4e-4f6e5757610b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:23.208194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:23.294878 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 672/1339] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:12:23 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:12:23.582135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede3600a-fcee-48c9-82cd-4922b1b0e35c None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:12:23.583662 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ede3600a-fcee-48c9-82cd-4922b1b0e35c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:23.692167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede3600a-fcee-48c9-82cd-4922b1b0e35c None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:12:23.763916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75270c5d-47de-46c5-9db5-d6ea21ca26d0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:12:23.764508 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75270c5d-47de-46c5-9db5-d6ea21ca26d0 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:23.764780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75270c5d-47de-46c5-9db5-d6ea21ca26d0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:23.765046 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75270c5d-47de-46c5-9db5-d6ea21ca26d0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:23.766634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-75270c5d-47de-46c5-9db5-d6ea21ca26d0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:23.801004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:23.873378 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75270c5d-47de-46c5-9db5-d6ea21ca26d0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:12:23.874822 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75270c5d-47de-46c5-9db5-d6ea21ca26d0 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:12:23.892358 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 668/1340] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:12:23 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:12:28.110689 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb53c6a-826b-456a-833d-4a6ddd1e70e7 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:28.111840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb53c6a-826b-456a-833d-4a6ddd1e70e7 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:28.112387 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb53c6a-826b-456a-833d-4a6ddd1e70e7 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:28.114454 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 673/1341] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:12:28 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:12:28.121610 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c05f432-f4d7-40d9-bdfb-beaf650e8667 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:28.122126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c05f432-f4d7-40d9-bdfb-beaf650e8667 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:28.122318 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c05f432-f4d7-40d9-bdfb-beaf650e8667 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:28.188803 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-7c05f432-f4d7-40d9-bdfb-beaf650e8667 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:12:28.190209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-7c05f432-f4d7-40d9-bdfb-beaf650e8667 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:28.284536 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:28.295734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 669/1342] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:12:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:12:28.583058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb53c6a-826b-456a-833d-4a6ddd1e70e7 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:12:28.584637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-9cb53c6a-826b-456a-833d-4a6ddd1e70e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:28.689956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb53c6a-826b-456a-833d-4a6ddd1e70e7 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:12:28.728378 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:28.764584 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22a89b8-7585-4e9f-ad4f-807e461bc48f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:12:28.765182 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22a89b8-7585-4e9f-ad4f-807e461bc48f service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:28.765458 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22a89b8-7585-4e9f-ad4f-807e461bc48f service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:28.765725 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22a89b8-7585-4e9f-ad4f-807e461bc48f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:28.767307 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-b22a89b8-7585-4e9f-ad4f-807e461bc48f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:28.873763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22a89b8-7585-4e9f-ad4f-807e461bc48f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:12:28.875197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22a89b8-7585-4e9f-ad4f-807e461bc48f service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:12:28.893032 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 674/1343] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:12:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:12:31.677802 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bf4588-3f9f-4f50-aa76-bf5d308c49fb None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:31.678851 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bf4588-3f9f-4f50-aa76-bf5d308c49fb None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:31.678851 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bf4588-3f9f-4f50-aa76-bf5d308c49fb None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:31.680206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 670/1344] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:12:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:12:31.685479 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d8e883-d142-42fa-b25c-07c3e1bf2ef8 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:31.685910 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d8e883-d142-42fa-b25c-07c3e1bf2ef8 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:31.686188 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d8e883-d142-42fa-b25c-07c3e1bf2ef8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:31.752306 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-48d8e883-d142-42fa-b25c-07c3e1bf2ef8 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:12:31.753525 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-48d8e883-d142-42fa-b25c-07c3e1bf2ef8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:31.773128 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:31.851720 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 675/1345] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:12:31 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:12:32.136889 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bf4588-3f9f-4f50-aa76-bf5d308c49fb None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:12:32.138804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-21bf4588-3f9f-4f50-aa76-bf5d308c49fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:32.247354 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bf4588-3f9f-4f50-aa76-bf5d308c49fb None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:12:32.320709 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9b3c8c-695c-4524-b795-9e80b2cfc709 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:12:32.321305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9b3c8c-695c-4524-b795-9e80b2cfc709 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:32.321576 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9b3c8c-695c-4524-b795-9e80b2cfc709 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:32.321918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9b3c8c-695c-4524-b795-9e80b2cfc709 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:32.323480 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-fe9b3c8c-695c-4524-b795-9e80b2cfc709 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:32.366389 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:32.436531 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9b3c8c-695c-4524-b795-9e80b2cfc709 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:12:32.438020 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9b3c8c-695c-4524-b795-9e80b2cfc709 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:12:32.456177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 671/1346] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:12:32 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:12:36.575400 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14122e04-e645-4d91-82ca-33d8f16b8d9d None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:36.575899 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14122e04-e645-4d91-82ca-33d8f16b8d9d None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:36.575996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14122e04-e645-4d91-82ca-33d8f16b8d9d None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:36.577330 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 676/1347] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:12:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:12:36.582678 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc77c1a1-7e70-4fc0-b696-5adf1cfb293f None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:36.583104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc77c1a1-7e70-4fc0-b696-5adf1cfb293f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:36.583245 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc77c1a1-7e70-4fc0-b696-5adf1cfb293f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:36.648569 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-bc77c1a1-7e70-4fc0-b696-5adf1cfb293f None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:12:36.649936 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-bc77c1a1-7e70-4fc0-b696-5adf1cfb293f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:36.748519 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:36.753886 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 672/1348] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:12:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:12:37.043455 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14122e04-e645-4d91-82ca-33d8f16b8d9d None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:12:37.044980 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-14122e04-e645-4d91-82ca-33d8f16b8d9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:37.148203 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14122e04-e645-4d91-82ca-33d8f16b8d9d None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:12:37.194692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:37.220297 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8e07b4-ceb9-4fbe-8b31-028da4a30886 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:12:37.220893 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8e07b4-ceb9-4fbe-8b31-028da4a30886 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:37.221170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8e07b4-ceb9-4fbe-8b31-028da4a30886 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:37.221440 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8e07b4-ceb9-4fbe-8b31-028da4a30886 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:37.223045 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-8d8e07b4-ceb9-4fbe-8b31-028da4a30886 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:37.328628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d8e07b4-ceb9-4fbe-8b31-028da4a30886 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:12:37.330103 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d8e07b4-ceb9-4fbe-8b31-028da4a30886 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:12:37.348146 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 677/1349] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:12:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:12:42.315349 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1425a170-59bf-4cc6-8f2e-02c631c06968 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:42.315784 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1425a170-59bf-4cc6-8f2e-02c631c06968 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:42.315995 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1425a170-59bf-4cc6-8f2e-02c631c06968 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:42.319368 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 673/1350] 10.222.0.26 () {54 vars in 921 bytes} [Tue Sep 27 04:12:42 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:12:42.324946 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7172b3-7ada-49e2-af84-bc94c5e56b1b None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:42.327595 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7172b3-7ada-49e2-af84-bc94c5e56b1b None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:42.327889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7172b3-7ada-49e2-af84-bc94c5e56b1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:42.460808 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-9f7172b3-7ada-49e2-af84-bc94c5e56b1b None None] MFA Rules not processed for user `3a1bc198dfc0486e9e56d4fb28cda05d`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:12:42.462208 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-9f7172b3-7ada-49e2-af84-bc94c5e56b1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:42.483441 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:42.586008 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 678/1351] 10.222.0.26 () {58 vars in 1018 bytes} [Tue Sep 27 04:12:42 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 262 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:12:46.742623 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385c72a2-59e4-4944-814b-d00209a8f4de None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:46.743167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385c72a2-59e4-4944-814b-d00209a8f4de None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:46.743167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385c72a2-59e4-4944-814b-d00209a8f4de None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:46.744703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 674/1352] 10.222.0.26 () {54 vars in 921 bytes} [Tue Sep 27 04:12:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:12:46.749598 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cd11ba-22cd-46b6-8ed8-2dd269080384 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:46.750094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cd11ba-22cd-46b6-8ed8-2dd269080384 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:46.750362 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cd11ba-22cd-46b6-8ed8-2dd269080384 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:46.830944 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-26cd11ba-22cd-46b6-8ed8-2dd269080384 None None] MFA Rules not processed for user `3a1bc198dfc0486e9e56d4fb28cda05d`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:12:46.832409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-26cd11ba-22cd-46b6-8ed8-2dd269080384 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:46.906637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:46.944822 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 679/1353] 10.222.0.26 () {58 vars in 1018 bytes} [Tue Sep 27 04:12:46 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:12:48.391470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c996ce2-6903-40a8-b002-ad8abc33fc3f None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:48.392005 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c996ce2-6903-40a8-b002-ad8abc33fc3f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:48.392005 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c996ce2-6903-40a8-b002-ad8abc33fc3f None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:48.393502 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 675/1354] 10.222.0.26 () {54 vars in 921 bytes} [Tue Sep 27 04:12:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:12:48.398427 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7492a8-010a-4a33-95e1-b8cf55fd165d None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:48.398822 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7492a8-010a-4a33-95e1-b8cf55fd165d None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:48.399103 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7492a8-010a-4a33-95e1-b8cf55fd165d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:48.477270 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-df7492a8-010a-4a33-95e1-b8cf55fd165d None None] MFA Rules not processed for user `3a1bc198dfc0486e9e56d4fb28cda05d`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:12:48.478509 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-df7492a8-010a-4a33-95e1-b8cf55fd165d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:48.585473 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 680/1355] 10.222.0.26 () {58 vars in 1018 bytes} [Tue Sep 27 04:12:48 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:12:52.083161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af76c7b4-0db1-41d0-b4dc-a451f9c8da9b None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:52.084024 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af76c7b4-0db1-41d0-b4dc-a451f9c8da9b None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:52.084486 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af76c7b4-0db1-41d0-b4dc-a451f9c8da9b None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:52.086025 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 676/1356] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:12:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:12:52.091856 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d5c13b-cf8b-4fd8-8eab-7097ffc7fedb None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:52.092357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d5c13b-cf8b-4fd8-8eab-7097ffc7fedb None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:52.092357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d5c13b-cf8b-4fd8-8eab-7097ffc7fedb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:52.132756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:52.166567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-05d5c13b-cf8b-4fd8-8eab-7097ffc7fedb None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:12:52.167762 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-05d5c13b-cf8b-4fd8-8eab-7097ffc7fedb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:52.277497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 681/1357] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:12:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:12:52.575241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367ae4c3-4c5c-4b6a-ac1f-52f050db57a6 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:52.575823 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367ae4c3-4c5c-4b6a-ac1f-52f050db57a6 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:52.576301 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367ae4c3-4c5c-4b6a-ac1f-52f050db57a6 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:52.577854 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 677/1358] 10.222.0.26 () {54 vars in 961 bytes} [Tue Sep 27 04:12:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:12:52.586562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72e93f4-f662-4464-945a-12e252ab138b None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:52.587224 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72e93f4-f662-4464-945a-12e252ab138b None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:52.587704 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72e93f4-f662-4464-945a-12e252ab138b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:52.683785 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-b72e93f4-f662-4464-945a-12e252ab138b None None] MFA Rules not processed for user `a1586ee841cd44658e4d5feb25de7940`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:12:52.685052 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-b72e93f4-f662-4464-945a-12e252ab138b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:52.740224 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:52.819561 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 682/1359] 10.222.0.26 () {58 vars in 1058 bytes} [Tue Sep 27 04:12:52 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:12:52.820721 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367ae4c3-4c5c-4b6a-ac1f-52f050db57a6 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:12:52.822260 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-367ae4c3-4c5c-4b6a-ac1f-52f050db57a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:52.979269 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367ae4c3-4c5c-4b6a-ac1f-52f050db57a6 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:12:53.062736 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:53.108190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ada462-70d9-4ec5-a1a5-a8a45538517f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:12:53.108980 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ada462-70d9-4ec5-a1a5-a8a45538517f service nova] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:53.109066 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ada462-70d9-4ec5-a1a5-a8a45538517f service nova] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:53.109437 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ada462-70d9-4ec5-a1a5-a8a45538517f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:53.110934 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-b6ada462-70d9-4ec5-a1a5-a8a45538517f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:53.216003 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ada462-70d9-4ec5-a1a5-a8a45538517f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:12:53.218093 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ada462-70d9-4ec5-a1a5-a8a45538517f service nova] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:12:53.237799 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 678/1360] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:12:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:12:55.617733 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf252b86-e4b3-4d58-8dfe-a0967a95a5b2 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:55.618323 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf252b86-e4b3-4d58-8dfe-a0967a95a5b2 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:55.618323 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf252b86-e4b3-4d58-8dfe-a0967a95a5b2 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:55.619941 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 683/1361] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:12:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:12:55.624994 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e5d657-385e-4908-886a-6756908583e2 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:55.624994 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e5d657-385e-4908-886a-6756908583e2 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:55.625881 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e5d657-385e-4908-886a-6756908583e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:55.700805 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-92e5d657-385e-4908-886a-6756908583e2 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:12:55.702040 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-92e5d657-385e-4908-886a-6756908583e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:55.745618 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:55.808388 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 679/1362] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:12:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:12:56.103710 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf252b86-e4b3-4d58-8dfe-a0967a95a5b2 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:12:56.105203 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-bf252b86-e4b3-4d58-8dfe-a0967a95a5b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:56.279515 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf252b86-e4b3-4d58-8dfe-a0967a95a5b2 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:12:56.291908 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:56.411874 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe79218-efa4-456c-bdbd-bc6442749d6a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:12:56.412476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe79218-efa4-456c-bdbd-bc6442749d6a service nova] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:56.412749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe79218-efa4-456c-bdbd-bc6442749d6a service nova] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:56.413018 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe79218-efa4-456c-bdbd-bc6442749d6a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:56.414698 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-4fe79218-efa4-456c-bdbd-bc6442749d6a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:56.509661 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe79218-efa4-456c-bdbd-bc6442749d6a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:12:56.511339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe79218-efa4-456c-bdbd-bc6442749d6a service nova] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:12:56.529311 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 684/1363] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:12:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:12:59.058783 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ca3412-e609-44cb-98bc-cd42556a3c12 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:59.059363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ca3412-e609-44cb-98bc-cd42556a3c12 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:59.059363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ca3412-e609-44cb-98bc-cd42556a3c12 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:59.060799 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 680/1364] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:12:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:12:59.065889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1818e685-8782-41af-9991-e3a40babd467 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:59.066309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1818e685-8782-41af-9991-e3a40babd467 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:59.066479 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1818e685-8782-41af-9991-e3a40babd467 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:59.081548 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:59.144328 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-1818e685-8782-41af-9991-e3a40babd467 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:12:59.147263 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-1818e685-8782-41af-9991-e3a40babd467 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:59.257162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 685/1365] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:12:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:12:59.552873 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690dc0c9-45b6-468b-be0d-3647d973a211 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:59.553176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690dc0c9-45b6-468b-be0d-3647d973a211 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:59.553451 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690dc0c9-45b6-468b-be0d-3647d973a211 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:59.554998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 681/1366] 10.222.0.26 () {54 vars in 961 bytes} [Tue Sep 27 04:12:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:12:59.560578 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7f4885-df9f-4383-b432-91b6ba3a9c10 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:12:59.560969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7f4885-df9f-4383-b432-91b6ba3a9c10 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:12:59.561311 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7f4885-df9f-4383-b432-91b6ba3a9c10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:12:59.641204 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-eb7f4885-df9f-4383-b432-91b6ba3a9c10 None None] MFA Rules not processed for user `a1586ee841cd44658e4d5feb25de7940`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:12:59.642430 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-eb7f4885-df9f-4383-b432-91b6ba3a9c10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:59.679004 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:12:59.764622 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 686/1367] 10.222.0.26 () {58 vars in 1058 bytes} [Tue Sep 27 04:12:59 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:12:59.773012 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690dc0c9-45b6-468b-be0d-3647d973a211 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:12:59.774626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-690dc0c9-45b6-468b-be0d-3647d973a211 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:12:59.930481 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690dc0c9-45b6-468b-be0d-3647d973a211 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:12:59.955208 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:00.058984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad94f95f-f5ac-4fed-bde8-37b1e8db0c4c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:13:00.059630 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad94f95f-f5ac-4fed-bde8-37b1e8db0c4c service nova] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:00.059869 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad94f95f-f5ac-4fed-bde8-37b1e8db0c4c service nova] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:00.060136 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad94f95f-f5ac-4fed-bde8-37b1e8db0c4c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:00.061691 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ad94f95f-f5ac-4fed-bde8-37b1e8db0c4c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:00.165850 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad94f95f-f5ac-4fed-bde8-37b1e8db0c4c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:13:00.167441 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad94f95f-f5ac-4fed-bde8-37b1e8db0c4c service nova] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:13:00.186884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 682/1368] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:12:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:13:00.226306 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be56dc1-16e8-47c9-a5ef-7c4d84a55d22 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:00.226848 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be56dc1-16e8-47c9-a5ef-7c4d84a55d22 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:00.226848 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be56dc1-16e8-47c9-a5ef-7c4d84a55d22 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:00.228310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 687/1369] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:13:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:13:00.237993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85a080d-d346-4775-a392-b93f2d024af8 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:00.237993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85a080d-d346-4775-a392-b93f2d024af8 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:00.237993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85a080d-d346-4775-a392-b93f2d024af8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:00.270364 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:00.318390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-b85a080d-d346-4775-a392-b93f2d024af8 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:13:00.319810 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-b85a080d-d346-4775-a392-b93f2d024af8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:00.446405 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 683/1370] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:13:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:13:00.774415 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0ef096-d3ed-4c89-baf6-efb5ecafc26a None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:00.774697 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0ef096-d3ed-4c89-baf6-efb5ecafc26a None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:00.774967 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0ef096-d3ed-4c89-baf6-efb5ecafc26a None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:00.776410 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 688/1371] 10.222.0.26 () {54 vars in 963 bytes} [Tue Sep 27 04:13:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:13:00.781212 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73dea9f6-7cf1-4ea1-a61b-9a40b31f11b0 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:00.781497 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73dea9f6-7cf1-4ea1-a61b-9a40b31f11b0 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:00.781792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73dea9f6-7cf1-4ea1-a61b-9a40b31f11b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:00.860847 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-73dea9f6-7cf1-4ea1-a61b-9a40b31f11b0 None None] MFA Rules not processed for user `3970310fc7e34704b686e8d3bff3fdb2`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:13:00.862312 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-73dea9f6-7cf1-4ea1-a61b-9a40b31f11b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:00.911651 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:00.944137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 684/1372] 10.222.0.26 () {58 vars in 1060 bytes} [Tue Sep 27 04:13:00 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:13:00.951779 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0ef096-d3ed-4c89-baf6-efb5ecafc26a None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:13:00.953264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-4a0ef096-d3ed-4c89-baf6-efb5ecafc26a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:01.065165 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0ef096-d3ed-4c89-baf6-efb5ecafc26a None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:13:01.112709 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:01.149277 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bfe50a-4e97-494b-8e44-bba096e0b972 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:13:01.149918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bfe50a-4e97-494b-8e44-bba096e0b972 service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:01.150246 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bfe50a-4e97-494b-8e44-bba096e0b972 service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:01.150520 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bfe50a-4e97-494b-8e44-bba096e0b972 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:01.152118 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-a2bfe50a-4e97-494b-8e44-bba096e0b972 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:01.291649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2bfe50a-4e97-494b-8e44-bba096e0b972 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:13:01.292954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2bfe50a-4e97-494b-8e44-bba096e0b972 service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:13:01.316121 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 689/1373] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:13:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:13:02.447428 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18dddea-c74a-430b-a2d2-614a33ba6617 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:02.447977 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18dddea-c74a-430b-a2d2-614a33ba6617 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:02.447977 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18dddea-c74a-430b-a2d2-614a33ba6617 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:02.449319 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 685/1374] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:13:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:13:02.454523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb84348-4802-4e12-8b1a-b3c804333371 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:02.454945 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb84348-4802-4e12-8b1a-b3c804333371 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:02.455061 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb84348-4802-4e12-8b1a-b3c804333371 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:02.523504 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-2bb84348-4802-4e12-8b1a-b3c804333371 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:13:02.525346 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-2bb84348-4802-4e12-8b1a-b3c804333371 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:02.546045 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:02.622922 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 690/1375] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:13:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:13:02.915894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18dddea-c74a-430b-a2d2-614a33ba6617 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:13:02.917516 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c18dddea-c74a-430b-a2d2-614a33ba6617 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:03.072328 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18dddea-c74a-430b-a2d2-614a33ba6617 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:13:03.124908 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:03.191623 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143c0100-d4b0-4009-b82f-5a15e87b323c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:13:03.192234 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143c0100-d4b0-4009-b82f-5a15e87b323c service nova] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:03.192509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143c0100-d4b0-4009-b82f-5a15e87b323c service nova] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:03.192779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143c0100-d4b0-4009-b82f-5a15e87b323c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:03.195063 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-143c0100-d4b0-4009-b82f-5a15e87b323c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:03.289171 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143c0100-d4b0-4009-b82f-5a15e87b323c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:13:03.290832 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143c0100-d4b0-4009-b82f-5a15e87b323c service nova] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:13:03.309895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 686/1376] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:13:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:13:05.529774 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a80ef9-7afe-428d-ae12-5341697fad7f None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:05.530365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a80ef9-7afe-428d-ae12-5341697fad7f None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:05.530365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a80ef9-7afe-428d-ae12-5341697fad7f None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:05.531801 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 691/1377] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:13:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:13:05.537062 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9016f8f-7e29-4924-8014-029fb0823550 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:05.538134 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9016f8f-7e29-4924-8014-029fb0823550 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:05.538616 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9016f8f-7e29-4924-8014-029fb0823550 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:05.607461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-e9016f8f-7e29-4924-8014-029fb0823550 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:13:05.608838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-e9016f8f-7e29-4924-8014-029fb0823550 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:05.616534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:05.717866 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 687/1378] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:13:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:13:06.000488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a80ef9-7afe-428d-ae12-5341697fad7f None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:13:06.002002 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-40a80ef9-7afe-428d-ae12-5341697fad7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:06.157535 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a80ef9-7afe-428d-ae12-5341697fad7f None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:13:06.161420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:06.273712 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f12c60-6bdd-4a66-b740-64ff85ebb2f5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:13:06.274246 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f12c60-6bdd-4a66-b740-64ff85ebb2f5 service nova] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:06.274519 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f12c60-6bdd-4a66-b740-64ff85ebb2f5 service nova] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:06.274783 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f12c60-6bdd-4a66-b740-64ff85ebb2f5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:06.276345 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-31f12c60-6bdd-4a66-b740-64ff85ebb2f5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:06.368657 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f12c60-6bdd-4a66-b740-64ff85ebb2f5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:13:06.370361 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f12c60-6bdd-4a66-b740-64ff85ebb2f5 service nova] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:13:06.388683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 692/1379] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:13:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 388 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:13:08.604707 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df03d3f-6868-4523-8ae1-5399e4db2b5a None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:08.605180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df03d3f-6868-4523-8ae1-5399e4db2b5a None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:08.605359 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df03d3f-6868-4523-8ae1-5399e4db2b5a None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:08.609780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 688/1380] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:13:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:13:08.612103 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d57ee7-781c-4f56-a80d-c3f68fab2c79 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:08.612525 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d57ee7-781c-4f56-a80d-c3f68fab2c79 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:08.612656 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d57ee7-781c-4f56-a80d-c3f68fab2c79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:08.625273 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:08.689835 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-15d57ee7-781c-4f56-a80d-c3f68fab2c79 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:13:08.692233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-15d57ee7-781c-4f56-a80d-c3f68fab2c79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:08.820198 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 693/1381] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:13:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:13:09.110812 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df03d3f-6868-4523-8ae1-5399e4db2b5a None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:13:09.112413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-7df03d3f-6868-4523-8ae1-5399e4db2b5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:09.262664 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:09.276868 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df03d3f-6868-4523-8ae1-5399e4db2b5a None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:13:09.395691 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a603072-be7a-488d-a244-753df1c16e82 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:13:09.396209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a603072-be7a-488d-a244-753df1c16e82 service nova] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:09.396487 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a603072-be7a-488d-a244-753df1c16e82 service nova] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:09.397330 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a603072-be7a-488d-a244-753df1c16e82 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:09.398974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-8a603072-be7a-488d-a244-753df1c16e82 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:09.494348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a603072-be7a-488d-a244-753df1c16e82 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:13:09.496005 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a603072-be7a-488d-a244-753df1c16e82 service nova] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:13:09.506241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:09.516626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 689/1382] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:13:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:13:12.242159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575e5174-7d52-4528-9a93-90f96f8d6112 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:12.242724 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575e5174-7d52-4528-9a93-90f96f8d6112 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:12.242822 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575e5174-7d52-4528-9a93-90f96f8d6112 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:12.244508 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 694/1383] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:13:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:13:12.250755 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea49545-a468-47b9-a71c-3e8e216d64b1 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:12.251230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea49545-a468-47b9-a71c-3e8e216d64b1 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:12.251342 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea49545-a468-47b9-a71c-3e8e216d64b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:12.318114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-bea49545-a468-47b9-a71c-3e8e216d64b1 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:13:12.319885 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-bea49545-a468-47b9-a71c-3e8e216d64b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:12.421256 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 690/1384] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:13:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:13:12.715410 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575e5174-7d52-4528-9a93-90f96f8d6112 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:13:12.717079 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-575e5174-7d52-4528-9a93-90f96f8d6112 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:12.804821 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:12.867139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575e5174-7d52-4528-9a93-90f96f8d6112 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:13:12.989997 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea699b8-ce0a-400f-b8d5-3095be35d9b0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:13:12.991107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea699b8-ce0a-400f-b8d5-3095be35d9b0 service nova] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:12.991382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea699b8-ce0a-400f-b8d5-3095be35d9b0 service nova] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:12.991653 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea699b8-ce0a-400f-b8d5-3095be35d9b0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:12.993472 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-eea699b8-ce0a-400f-b8d5-3095be35d9b0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:13.051019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:13.096665 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea699b8-ce0a-400f-b8d5-3095be35d9b0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:13:13.098708 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea699b8-ce0a-400f-b8d5-3095be35d9b0 service nova] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:13:13.123276 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 695/1385] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:13:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:13:15.325700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b561b6c-51ff-43c4-a2b1-6373b44889c6 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:15.326273 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b561b6c-51ff-43c4-a2b1-6373b44889c6 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:15.326273 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b561b6c-51ff-43c4-a2b1-6373b44889c6 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:15.327613 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 691/1386] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:13:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:13:15.333796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fe98c3-535b-450b-ab22-0ecb3228705f None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:15.334277 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fe98c3-535b-450b-ab22-0ecb3228705f None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:15.334451 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fe98c3-535b-450b-ab22-0ecb3228705f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:15.410823 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-05fe98c3-535b-450b-ab22-0ecb3228705f None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:13:15.412105 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-05fe98c3-535b-450b-ab22-0ecb3228705f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:15.513391 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 696/1387] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:13:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:13:15.805028 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b561b6c-51ff-43c4-a2b1-6373b44889c6 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:13:15.806567 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-2b561b6c-51ff-43c4-a2b1-6373b44889c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:15.882693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:15.957156 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b561b6c-51ff-43c4-a2b1-6373b44889c6 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:13:16.076818 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e20b77-0a12-4f68-b6f1-f8470206d255 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:13:16.077340 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e20b77-0a12-4f68-b6f1-f8470206d255 service nova] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:16.077615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e20b77-0a12-4f68-b6f1-f8470206d255 service nova] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:16.077919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e20b77-0a12-4f68-b6f1-f8470206d255 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:16.079500 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a1e20b77-0a12-4f68-b6f1-f8470206d255 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:16.119616 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:16.171041 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e20b77-0a12-4f68-b6f1-f8470206d255 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:13:16.172646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e20b77-0a12-4f68-b6f1-f8470206d255 service nova] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:13:16.191073 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 692/1388] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:13:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:13:17.584703 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a312d12c-0f28-4d6f-8722-c89c6258f389 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:17.585260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a312d12c-0f28-4d6f-8722-c89c6258f389 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:17.585260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a312d12c-0f28-4d6f-8722-c89c6258f389 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:17.586610 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 697/1389] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:13:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:13:17.592016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6033a385-497b-4d78-bed6-2426fc62617a None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:17.592442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6033a385-497b-4d78-bed6-2426fc62617a None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:17.592602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6033a385-497b-4d78-bed6-2426fc62617a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:17.657796 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-6033a385-497b-4d78-bed6-2426fc62617a None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:13:17.658904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6033a385-497b-4d78-bed6-2426fc62617a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:17.754285 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 693/1390] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:13:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:13:18.059712 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1ae97b-07f5-4a95-9139-d2fef266ac2e None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:18.060020 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1ae97b-07f5-4a95-9139-d2fef266ac2e None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:18.060297 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1ae97b-07f5-4a95-9139-d2fef266ac2e None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:18.061611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 698/1391] 10.222.0.26 () {54 vars in 962 bytes} [Tue Sep 27 04:13:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:13:18.067024 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008d0f96-bcaf-4668-9f0c-e7e74d84c6a2 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:18.067306 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008d0f96-bcaf-4668-9f0c-e7e74d84c6a2 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:18.067578 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008d0f96-bcaf-4668-9f0c-e7e74d84c6a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:18.083244 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:18.147855 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-008d0f96-bcaf-4668-9f0c-e7e74d84c6a2 None None] MFA Rules not processed for user `8dcc53332e6b462baf349a72b19d2b3b`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:13:18.149094 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-008d0f96-bcaf-4668-9f0c-e7e74d84c6a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:18.232987 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 694/1392] 10.222.0.26 () {58 vars in 1059 bytes} [Tue Sep 27 04:13:18 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:13:18.242065 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1ae97b-07f5-4a95-9139-d2fef266ac2e None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:13:18.243568 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ad1ae97b-07f5-4a95-9139-d2fef266ac2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:18.277844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:18.353599 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1ae97b-07f5-4a95-9139-d2fef266ac2e None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:13:18.384667 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b325589-0610-4291-b535-4e5da8a2a6a8 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:18.384950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b325589-0610-4291-b535-4e5da8a2a6a8 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:18.385226 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b325589-0610-4291-b535-4e5da8a2a6a8 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:18.386602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 695/1393] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:13:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:13:18.391706 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586631e5-df9c-4438-a920-ca010927ce18 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:18.391994 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586631e5-df9c-4438-a920-ca010927ce18 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:18.392275 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586631e5-df9c-4438-a920-ca010927ce18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:18.437241 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bfa4ed-b44e-4a96-bf7d-ce989407f8e7 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dcc53332e6b462baf349a72b19d2b3b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:13:18.437871 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bfa4ed-b44e-4a96-bf7d-ce989407f8e7 service glance] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:18.438148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bfa4ed-b44e-4a96-bf7d-ce989407f8e7 service glance] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:18.438424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bfa4ed-b44e-4a96-bf7d-ce989407f8e7 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:18.440020 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-82bfa4ed-b44e-4a96-bf7d-ce989407f8e7 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:18.463469 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-586631e5-df9c-4438-a920-ca010927ce18 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:13:18.464693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-586631e5-df9c-4438-a920-ca010927ce18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:18.519601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:18.520219 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:18.553683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bfa4ed-b44e-4a96-bf7d-ce989407f8e7 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:13:18.555124 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bfa4ed-b44e-4a96-bf7d-ce989407f8e7 service glance] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:13:18.562599 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 696/1394] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:13:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:13:18.574232 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 699/1395] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:13:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:13:18.852492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586631e5-df9c-4438-a920-ca010927ce18 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:13:18.853678 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-586631e5-df9c-4438-a920-ca010927ce18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:18.977428 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586631e5-df9c-4438-a920-ca010927ce18 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:13:19.043478 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:19.090325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452e274e-8955-4585-9cc2-100205a8c322 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:13:19.090958 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452e274e-8955-4585-9cc2-100205a8c322 service nova] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:19.091230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452e274e-8955-4585-9cc2-100205a8c322 service nova] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:19.091494 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452e274e-8955-4585-9cc2-100205a8c322 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:19.093071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-452e274e-8955-4585-9cc2-100205a8c322 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:19.190687 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452e274e-8955-4585-9cc2-100205a8c322 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:13:19.192309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452e274e-8955-4585-9cc2-100205a8c322 service nova] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:13:19.218303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 697/1396] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:13:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:13:19.320174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81814cfe-d00a-43f7-b747-43eee6789683 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:19.320743 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81814cfe-d00a-43f7-b747-43eee6789683 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:19.320743 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81814cfe-d00a-43f7-b747-43eee6789683 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:19.395113 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-81814cfe-d00a-43f7-b747-43eee6789683 None None] MFA Rules not processed for user `5a2fcc8c9ac44dceadcc30d163bbfa0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:13:19.396356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-81814cfe-d00a-43f7-b747-43eee6789683 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:19.498881 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 700/1397] 10.222.0.26 () {58 vars in 1009 bytes} [Tue Sep 27 04:13:19 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:13:19.541155 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4de04c-25c7-44fe-82f6-c4d288ce3114 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:19.541528 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4de04c-25c7-44fe-82f6-c4d288ce3114 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:19.541903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4de04c-25c7-44fe-82f6-c4d288ce3114 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:19.543530 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 698/1398] 10.222.0.26 () {54 vars in 943 bytes} [Tue Sep 27 04:13:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:13:19.549405 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ddb64f-44c2-429c-8b0c-b9153152158b None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:19.549795 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ddb64f-44c2-429c-8b0c-b9153152158b None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:19.550155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ddb64f-44c2-429c-8b0c-b9153152158b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:19.607485 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:19.627177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-67ddb64f-44c2-429c-8b0c-b9153152158b None None] MFA Rules not processed for user `181dd5dc720c4a97860852062b7b19c3`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:13:19.628380 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-67ddb64f-44c2-429c-8b0c-b9153152158b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:19.735042 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 701/1399] 10.222.0.26 () {58 vars in 1040 bytes} [Tue Sep 27 04:13:19 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:13:19.744590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4de04c-25c7-44fe-82f6-c4d288ce3114 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:13:19.746292 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-7c4de04c-25c7-44fe-82f6-c4d288ce3114 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:19.826133 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:19.882527 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4de04c-25c7-44fe-82f6-c4d288ce3114 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:13:19.986269 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa571e7e-65a2-4ddf-a77c-db1d19647f73 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '181dd5dc720c4a97860852062b7b19c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:13:19.986908 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa571e7e-65a2-4ddf-a77c-db1d19647f73 service swift] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:19.987190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa571e7e-65a2-4ddf-a77c-db1d19647f73 service swift] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:19.987461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa571e7e-65a2-4ddf-a77c-db1d19647f73 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:19.989116 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-fa571e7e-65a2-4ddf-a77c-db1d19647f73 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:20.053007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:20.072555 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa571e7e-65a2-4ddf-a77c-db1d19647f73 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:13:20.074175 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa571e7e-65a2-4ddf-a77c-db1d19647f73 service swift] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:13:20.082014 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 699/1400] 10.222.0.26 () {60 vars in 1384 bytes} [Tue Sep 27 04:13:19 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 27 04:13:21.433717 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d2f823-cca7-44ff-9315-86dbc0b7c756 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:21.435040 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d2f823-cca7-44ff-9315-86dbc0b7c756 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:21.435325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d2f823-cca7-44ff-9315-86dbc0b7c756 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:21.436834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 702/1401] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:13:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:13:21.442072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab839c0-153a-4228-b034-c4542b260f30 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:21.442524 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab839c0-153a-4228-b034-c4542b260f30 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:21.442613 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab839c0-153a-4228-b034-c4542b260f30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:21.516574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-3ab839c0-153a-4228-b034-c4542b260f30 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:13:21.517816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-3ab839c0-153a-4228-b034-c4542b260f30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:21.626461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 700/1402] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:13:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:13:21.916065 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d2f823-cca7-44ff-9315-86dbc0b7c756 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:13:21.917656 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-69d2f823-cca7-44ff-9315-86dbc0b7c756 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:22.034585 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:22.082139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d2f823-cca7-44ff-9315-86dbc0b7c756 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:13:22.216440 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f853de2-7da8-46af-aef0-087459534ec9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:13:22.217225 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f853de2-7da8-46af-aef0-087459534ec9 service nova] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:22.217509 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f853de2-7da8-46af-aef0-087459534ec9 service nova] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:22.217814 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f853de2-7da8-46af-aef0-087459534ec9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:22.219427 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-9f853de2-7da8-46af-aef0-087459534ec9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:22.299025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:22.325555 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f853de2-7da8-46af-aef0-087459534ec9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:13:22.327525 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f853de2-7da8-46af-aef0-087459534ec9 service nova] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:13:22.348825 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 703/1403] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:13:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:13:24.577268 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fed060b-9314-4c60-bbed-592d5df5e65b None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:24.577859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fed060b-9314-4c60-bbed-592d5df5e65b None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:24.577859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fed060b-9314-4c60-bbed-592d5df5e65b None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:24.579170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 701/1404] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:13:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:13:24.584963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08de5f6-4bae-4c90-9ff1-069a304b6308 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:24.585514 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08de5f6-4bae-4c90-9ff1-069a304b6308 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:24.585514 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08de5f6-4bae-4c90-9ff1-069a304b6308 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:24.656797 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-a08de5f6-4bae-4c90-9ff1-069a304b6308 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:13:24.658291 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-a08de5f6-4bae-4c90-9ff1-069a304b6308 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:24.766565 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 704/1405] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:13:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:13:25.058637 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fed060b-9314-4c60-bbed-592d5df5e65b None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:13:25.060516 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-9fed060b-9314-4c60-bbed-592d5df5e65b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:25.216526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:25.314622 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fed060b-9314-4c60-bbed-592d5df5e65b None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:13:25.445552 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba48da66-6618-4dab-90f8-a89540b7a7be service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:13:25.446317 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba48da66-6618-4dab-90f8-a89540b7a7be service nova] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:25.446594 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba48da66-6618-4dab-90f8-a89540b7a7be service nova] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:25.446865 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba48da66-6618-4dab-90f8-a89540b7a7be service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:25.448444 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ba48da66-6618-4dab-90f8-a89540b7a7be service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:25.471934 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:25.551279 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba48da66-6618-4dab-90f8-a89540b7a7be service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:13:25.552878 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba48da66-6618-4dab-90f8-a89540b7a7be service nova] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:13:25.572783 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 702/1406] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:13:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 514 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:13:27.791820 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4155288-c52c-4d6b-ae1f-23ada5296b68 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:27.792412 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4155288-c52c-4d6b-ae1f-23ada5296b68 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:27.792412 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4155288-c52c-4d6b-ae1f-23ada5296b68 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:27.793680 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 705/1407] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:13:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:13:27.799255 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b065535b-acca-45b2-9e51-f074ab3484f3 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:27.800066 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b065535b-acca-45b2-9e51-f074ab3484f3 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:27.800344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b065535b-acca-45b2-9e51-f074ab3484f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:27.874145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-b065535b-acca-45b2-9e51-f074ab3484f3 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:13:27.875472 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-b065535b-acca-45b2-9e51-f074ab3484f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:27.972197 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:27.993769 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 703/1408] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:13:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:13:28.281459 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4155288-c52c-4d6b-ae1f-23ada5296b68 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:13:28.283223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-e4155288-c52c-4d6b-ae1f-23ada5296b68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:28.338383 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:28.449584 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4155288-c52c-4d6b-ae1f-23ada5296b68 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:13:28.580494 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b9742a-7c38-4962-b5bb-968a1d52b54d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:13:28.581166 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b9742a-7c38-4962-b5bb-968a1d52b54d service nova] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:28.581523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b9742a-7c38-4962-b5bb-968a1d52b54d service nova] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:28.581920 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b9742a-7c38-4962-b5bb-968a1d52b54d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:28.583520 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-15b9742a-7c38-4962-b5bb-968a1d52b54d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:28.592573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:28.684037 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b9742a-7c38-4962-b5bb-968a1d52b54d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:13:28.685776 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b9742a-7c38-4962-b5bb-968a1d52b54d service nova] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:13:28.705849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 706/1409] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:13:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:13:37.977362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:38.703971 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=42 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:42.773718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d2b3fd-044f-4426-abf6-e6b8ef9ecf44 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:42.774265 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d2b3fd-044f-4426-abf6-e6b8ef9ecf44 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:42.774370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d2b3fd-044f-4426-abf6-e6b8ef9ecf44 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:42.775632 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 704/1410] 10.222.0.26 () {54 vars in 918 bytes} [Tue Sep 27 04:13:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:13:42.780696 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c2ffbf-e346-48b4-89bd-934631b966c7 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:42.781135 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c2ffbf-e346-48b4-89bd-934631b966c7 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:42.781242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c2ffbf-e346-48b4-89bd-934631b966c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:42.856345 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-b7c2ffbf-e346-48b4-89bd-934631b966c7 None None] MFA Rules not processed for user `3a1bc198dfc0486e9e56d4fb28cda05d`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:13:42.857538 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-b7c2ffbf-e346-48b4-89bd-934631b966c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:42.959247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 707/1411] 10.222.0.26 () {58 vars in 1015 bytes} [Tue Sep 27 04:13:42 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:13:42.973402 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a138bf0-84de-4696-bc61-86cbfef5850d None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:42.973685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a138bf0-84de-4696-bc61-86cbfef5850d None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:42.974001 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a138bf0-84de-4696-bc61-86cbfef5850d None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:42.975316 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 705/1412] 10.222.0.26 () {54 vars in 918 bytes} [Tue Sep 27 04:13:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:13:42.979934 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077f622c-e6b5-4a2c-9d06-af054d249d2c None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:42.980217 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077f622c-e6b5-4a2c-9d06-af054d249d2c None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:42.980489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077f622c-e6b5-4a2c-9d06-af054d249d2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:43.051970 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:43.055680 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-077f622c-e6b5-4a2c-9d06-af054d249d2c None None] MFA Rules not processed for user `a1586ee841cd44658e4d5feb25de7940`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:13:43.057468 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-077f622c-e6b5-4a2c-9d06-af054d249d2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:43.170899 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 708/1413] 10.222.0.26 () {58 vars in 1015 bytes} [Tue Sep 27 04:13:42 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:13:45.732683 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b273909e-a6d6-4f70-8350-aed052fe27c5 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:45.733231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b273909e-a6d6-4f70-8350-aed052fe27c5 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:45.733231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b273909e-a6d6-4f70-8350-aed052fe27c5 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:45.734580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 706/1414] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 04:13:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:13:45.740306 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d9a8d6-4315-4220-873e-e926d404741f None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:45.741275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d9a8d6-4315-4220-873e-e926d404741f None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:45.741377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d9a8d6-4315-4220-873e-e926d404741f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:45.805609 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-94d9a8d6-4315-4220-873e-e926d404741f None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:13:45.806831 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-94d9a8d6-4315-4220-873e-e926d404741f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:45.859847 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:45.901597 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 709/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 04:13:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:13:46.191296 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b273909e-a6d6-4f70-8350-aed052fe27c5 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:13:46.192853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-b273909e-a6d6-4f70-8350-aed052fe27c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:46.335792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b273909e-a6d6-4f70-8350-aed052fe27c5 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:13:46.407065 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:46.446752 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74a7c58-d870-4281-82e4-f15a1471614e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:13:46.447282 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74a7c58-d870-4281-82e4-f15a1471614e service nova] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:46.447553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74a7c58-d870-4281-82e4-f15a1471614e service nova] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:46.447819 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74a7c58-d870-4281-82e4-f15a1471614e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:46.449386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c74a7c58-d870-4281-82e4-f15a1471614e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:46.536602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74a7c58-d870-4281-82e4-f15a1471614e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:13:46.538808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74a7c58-d870-4281-82e4-f15a1471614e service nova] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:13:46.555910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 707/1416] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:13:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:13:47.694793 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffa57d5-5af6-4d1c-a3ba-245ef74f051c None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:47.695328 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffa57d5-5af6-4d1c-a3ba-245ef74f051c None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:47.695328 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffa57d5-5af6-4d1c-a3ba-245ef74f051c None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:47.696569 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 710/1417] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 27 04:13:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:13:47.702622 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8baedd-52cb-44e3-9e99-56ab857f587a None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:47.703078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8baedd-52cb-44e3-9e99-56ab857f587a None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:47.703185 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8baedd-52cb-44e3-9e99-56ab857f587a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:47.778972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-cf8baedd-52cb-44e3-9e99-56ab857f587a None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:13:47.780204 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-cf8baedd-52cb-44e3-9e99-56ab857f587a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:47.806596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:47.874784 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 708/1418] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 27 04:13:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:13:48.024576 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffa57d5-5af6-4d1c-a3ba-245ef74f051c None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:13:48.026114 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-fffa57d5-5af6-4d1c-a3ba-245ef74f051c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:48.171642 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffa57d5-5af6-4d1c-a3ba-245ef74f051c None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:13:48.211613 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:48.283073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58419078-00a4-4774-8c02-ae2629aff0d9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:13:48.283608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58419078-00a4-4774-8c02-ae2629aff0d9 service nova] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:48.283886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58419078-00a4-4774-8c02-ae2629aff0d9 service nova] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:48.284159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58419078-00a4-4774-8c02-ae2629aff0d9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:48.285774 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-58419078-00a4-4774-8c02-ae2629aff0d9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:48.373463 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58419078-00a4-4774-8c02-ae2629aff0d9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:13:48.375061 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58419078-00a4-4774-8c02-ae2629aff0d9 service nova] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:13:48.392847 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 711/1419] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:13:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:13:49.522541 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2179f0-c65d-452b-8ded-1203e5231df2 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:49.523118 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2179f0-c65d-452b-8ded-1203e5231df2 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:49.523118 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2179f0-c65d-452b-8ded-1203e5231df2 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:49.524402 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 709/1420] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 27 04:13:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:13:49.530476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfbeff9-18eb-4939-91b6-dc8e6c36e557 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:49.530896 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfbeff9-18eb-4939-91b6-dc8e6c36e557 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:49.531024 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfbeff9-18eb-4939-91b6-dc8e6c36e557 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:49.584276 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:49.606343 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-6cfbeff9-18eb-4939-91b6-dc8e6c36e557 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:13:49.607540 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-6cfbeff9-18eb-4939-91b6-dc8e6c36e557 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:49.703303 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 712/1421] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 27 04:13:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:13:49.851504 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2179f0-c65d-452b-8ded-1203e5231df2 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:13:49.853015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-2e2179f0-c65d-452b-8ded-1203e5231df2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:49.996610 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2179f0-c65d-452b-8ded-1203e5231df2 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:13:50.008374 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:50.106201 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b219bd1-2b5c-4033-a7b2-d88d78bd30b3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:13:50.106801 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b219bd1-2b5c-4033-a7b2-d88d78bd30b3 service nova] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:50.107071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b219bd1-2b5c-4033-a7b2-d88d78bd30b3 service nova] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:50.107334 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b219bd1-2b5c-4033-a7b2-d88d78bd30b3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:50.108927 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-9b219bd1-2b5c-4033-a7b2-d88d78bd30b3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:50.196355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b219bd1-2b5c-4033-a7b2-d88d78bd30b3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:13:50.197957 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b219bd1-2b5c-4033-a7b2-d88d78bd30b3 service nova] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:13:50.216321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 710/1422] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:13:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:13:51.315336 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11078c8c-6069-4b96-bd09-a9739d89045f None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:51.315885 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11078c8c-6069-4b96-bd09-a9739d89045f None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:51.315885 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11078c8c-6069-4b96-bd09-a9739d89045f None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:51.317185 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 713/1423] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 27 04:13:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:13:51.323281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c225584b-ba94-418e-a097-01be6883c8b9 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:51.323700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c225584b-ba94-418e-a097-01be6883c8b9 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:51.323832 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c225584b-ba94-418e-a097-01be6883c8b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:51.343312 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:51.399113 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-c225584b-ba94-418e-a097-01be6883c8b9 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:13:51.400311 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c225584b-ba94-418e-a097-01be6883c8b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:51.494360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 711/1424] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 27 04:13:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:13:51.646335 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11078c8c-6069-4b96-bd09-a9739d89045f None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:13:51.647776 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-11078c8c-6069-4b96-bd09-a9739d89045f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:51.754584 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:51.790447 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11078c8c-6069-4b96-bd09-a9739d89045f None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:13:51.901973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d934847-c0d0-4a91-9c91-6b501bc63476 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:13:51.902486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d934847-c0d0-4a91-9c91-6b501bc63476 service nova] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:51.902762 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d934847-c0d0-4a91-9c91-6b501bc63476 service nova] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:51.903030 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d934847-c0d0-4a91-9c91-6b501bc63476 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:51.904608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-7d934847-c0d0-4a91-9c91-6b501bc63476 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:51.981533 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:51.995119 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d934847-c0d0-4a91-9c91-6b501bc63476 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:13:51.996698 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d934847-c0d0-4a91-9c91-6b501bc63476 service nova] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:13:52.014379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 714/1425] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:13:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:13:53.114541 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f75fc5d-78ff-4276-a9fd-524c686ce596 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:53.115090 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f75fc5d-78ff-4276-a9fd-524c686ce596 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:53.115090 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f75fc5d-78ff-4276-a9fd-524c686ce596 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:53.116966 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 712/1426] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 27 04:13:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:13:53.123133 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11edc96e-a04c-4040-8d21-497fba7e6dc3 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:53.123564 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11edc96e-a04c-4040-8d21-497fba7e6dc3 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:53.123706 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11edc96e-a04c-4040-8d21-497fba7e6dc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:53.198347 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-11edc96e-a04c-4040-8d21-497fba7e6dc3 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:13:53.199564 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-11edc96e-a04c-4040-8d21-497fba7e6dc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:53.301060 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 715/1427] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 27 04:13:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:13:53.451336 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f75fc5d-78ff-4276-a9fd-524c686ce596 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:13:53.452844 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-5f75fc5d-78ff-4276-a9fd-524c686ce596 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:53.536348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:53.597761 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f75fc5d-78ff-4276-a9fd-524c686ce596 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:13:53.710022 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d4ce15-97ea-4933-a6b2-b1576ed253ac service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:13:53.711165 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d4ce15-97ea-4933-a6b2-b1576ed253ac service nova] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:53.711440 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d4ce15-97ea-4933-a6b2-b1576ed253ac service nova] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:53.711707 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d4ce15-97ea-4933-a6b2-b1576ed253ac service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:53.713314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-92d4ce15-97ea-4933-a6b2-b1576ed253ac service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:53.760933 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:53.801544 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d4ce15-97ea-4933-a6b2-b1576ed253ac service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:13:53.803169 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d4ce15-97ea-4933-a6b2-b1576ed253ac service nova] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:13:53.821018 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 713/1428] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:13:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:13:54.878422 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6df6aba-7e6a-4f8e-baed-c470481370c9 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:54.878954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6df6aba-7e6a-4f8e-baed-c470481370c9 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:54.878954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6df6aba-7e6a-4f8e-baed-c470481370c9 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:54.880322 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 716/1429] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 27 04:13:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:13:54.886399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7535906a-5090-4914-a08f-25f07e30b1a8 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:54.886835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7535906a-5090-4914-a08f-25f07e30b1a8 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:54.886930 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7535906a-5090-4914-a08f-25f07e30b1a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:54.961366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-7535906a-5090-4914-a08f-25f07e30b1a8 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:13:54.962589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-7535906a-5090-4914-a08f-25f07e30b1a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:55.059276 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 714/1430] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 27 04:13:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:13:55.212359 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6df6aba-7e6a-4f8e-baed-c470481370c9 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:13:55.213904 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c6df6aba-7e6a-4f8e-baed-c470481370c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:55.264901 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:55.359023 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6df6aba-7e6a-4f8e-baed-c470481370c9 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:13:55.472932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcdd548-68df-4f4d-bd2b-51febbbb326c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:13:55.473540 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcdd548-68df-4f4d-bd2b-51febbbb326c service nova] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:55.473848 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcdd548-68df-4f4d-bd2b-51febbbb326c service nova] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:55.474125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcdd548-68df-4f4d-bd2b-51febbbb326c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:55.475700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-3bcdd548-68df-4f4d-bd2b-51febbbb326c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:55.482547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:55.563894 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bcdd548-68df-4f4d-bd2b-51febbbb326c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:13:55.565477 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bcdd548-68df-4f4d-bd2b-51febbbb326c service nova] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:13:55.583444 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 717/1431] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:13:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:13:56.637146 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415bfdb7-1429-46a7-881a-a4937b8381f8 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:56.637798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415bfdb7-1429-46a7-881a-a4937b8381f8 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:56.637798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415bfdb7-1429-46a7-881a-a4937b8381f8 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:56.639048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 715/1432] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 27 04:13:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:13:56.645540 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7e07f2-3d76-4a1b-91cc-ffd11bf48798 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:56.646043 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7e07f2-3d76-4a1b-91cc-ffd11bf48798 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:56.646160 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7e07f2-3d76-4a1b-91cc-ffd11bf48798 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:56.721097 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-0c7e07f2-3d76-4a1b-91cc-ffd11bf48798 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:13:56.722253 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0c7e07f2-3d76-4a1b-91cc-ffd11bf48798 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:56.790314 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:56.818514 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 718/1433] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 27 04:13:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:13:56.967230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415bfdb7-1429-46a7-881a-a4937b8381f8 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:13:56.968735 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-415bfdb7-1429-46a7-881a-a4937b8381f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:56.985913 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:57.110654 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415bfdb7-1429-46a7-881a-a4937b8381f8 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:13:57.195387 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:57.219907 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2fc3772-191b-405b-8996-cb49a5944557 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:13:57.220503 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fc3772-191b-405b-8996-cb49a5944557 service nova] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:57.220778 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fc3772-191b-405b-8996-cb49a5944557 service nova] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:57.221046 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fc3772-191b-405b-8996-cb49a5944557 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:57.222637 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d2fc3772-191b-405b-8996-cb49a5944557 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:57.310768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2fc3772-191b-405b-8996-cb49a5944557 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:13:57.312265 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2fc3772-191b-405b-8996-cb49a5944557 service nova] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:13:57.329426 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 716/1434] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:13:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:13:58.400944 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7435398-d186-45c9-82dc-df7e929465db None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:58.401483 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7435398-d186-45c9-82dc-df7e929465db None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:58.401483 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7435398-d186-45c9-82dc-df7e929465db None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:58.402823 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 719/1435] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 27 04:13:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:13:58.409231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf623ee-bef3-40b8-bf36-820573f07e41 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:58.409672 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf623ee-bef3-40b8-bf36-820573f07e41 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:58.409818 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf623ee-bef3-40b8-bf36-820573f07e41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:58.483595 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-4cf623ee-bef3-40b8-bf36-820573f07e41 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:13:58.484778 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-4cf623ee-bef3-40b8-bf36-820573f07e41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:58.526705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:58.579926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 717/1436] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 27 04:13:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:13:58.729801 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7435398-d186-45c9-82dc-df7e929465db None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:13:58.731329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-f7435398-d186-45c9-82dc-df7e929465db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:58.874712 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7435398-d186-45c9-82dc-df7e929465db None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:13:58.926997 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:13:58.983988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09e847b-9eae-4ac5-9764-59335ee9822d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:13:58.984538 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09e847b-9eae-4ac5-9764-59335ee9822d service nova] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:13:58.984827 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09e847b-9eae-4ac5-9764-59335ee9822d service nova] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:13:58.985091 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09e847b-9eae-4ac5-9764-59335ee9822d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:13:58.986737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-d09e847b-9eae-4ac5-9764-59335ee9822d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:13:59.074329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09e847b-9eae-4ac5-9764-59335ee9822d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:13:59.075908 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09e847b-9eae-4ac5-9764-59335ee9822d service nova] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:13:59.093161 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 720/1437] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:13:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:14:00.164877 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2722f2-a185-49db-a882-b39519b5d160 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:00.165451 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2722f2-a185-49db-a882-b39519b5d160 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:00.165563 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2722f2-a185-49db-a882-b39519b5d160 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:00.166920 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 718/1438] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 27 04:14:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:14:00.173237 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e239f2-fbd2-4b5e-93ff-584f1402533b None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:00.173669 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e239f2-fbd2-4b5e-93ff-584f1402533b None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:00.173802 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e239f2-fbd2-4b5e-93ff-584f1402533b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:00.249674 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-55e239f2-fbd2-4b5e-93ff-584f1402533b None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:14:00.250911 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-55e239f2-fbd2-4b5e-93ff-584f1402533b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:00.252878 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:00.345984 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 721/1439] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 27 04:14:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:14:00.494829 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2722f2-a185-49db-a882-b39519b5d160 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:14:00.496349 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6d2722f2-a185-49db-a882-b39519b5d160 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:00.636288 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2722f2-a185-49db-a882-b39519b5d160 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:14:00.664926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:00.754661 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1fa23fe-9408-4f6d-8f8a-1bdff24daee6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:14:00.755300 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fa23fe-9408-4f6d-8f8a-1bdff24daee6 service nova] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:00.755573 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fa23fe-9408-4f6d-8f8a-1bdff24daee6 service nova] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:00.755840 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fa23fe-9408-4f6d-8f8a-1bdff24daee6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:00.757396 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a1fa23fe-9408-4f6d-8f8a-1bdff24daee6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:00.843846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1fa23fe-9408-4f6d-8f8a-1bdff24daee6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:14:00.845408 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1fa23fe-9408-4f6d-8f8a-1bdff24daee6 service nova] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:14:00.863281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 719/1440] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:14:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:14:02.911370 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ad2941-c85a-418d-adfd-40f2fe8679a5 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:02.911932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ad2941-c85a-418d-adfd-40f2fe8679a5 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:02.911932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ad2941-c85a-418d-adfd-40f2fe8679a5 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:02.913227 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 722/1441] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 04:14:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:14:02.918471 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede2a635-84c7-42c5-aac7-5419422c9d3b None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:02.918919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede2a635-84c7-42c5-aac7-5419422c9d3b None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:02.919030 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede2a635-84c7-42c5-aac7-5419422c9d3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:02.959847 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:02.984159 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-ede2a635-84c7-42c5-aac7-5419422c9d3b None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:14:02.985335 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ede2a635-84c7-42c5-aac7-5419422c9d3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:03.079715 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 720/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 04:14:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:14:03.367954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ad2941-c85a-418d-adfd-40f2fe8679a5 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:14:03.369464 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-00ad2941-c85a-418d-adfd-40f2fe8679a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:03.469928 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ad2941-c85a-418d-adfd-40f2fe8679a5 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:14:03.502593 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:03.543807 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c592a5f-666e-4ceb-a316-d10338c233a2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:14:03.544410 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c592a5f-666e-4ceb-a316-d10338c233a2 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:03.544689 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c592a5f-666e-4ceb-a316-d10338c233a2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:03.544962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c592a5f-666e-4ceb-a316-d10338c233a2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:03.546580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-1c592a5f-666e-4ceb-a316-d10338c233a2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:03.653044 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c592a5f-666e-4ceb-a316-d10338c233a2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:14:03.654516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c592a5f-666e-4ceb-a316-d10338c233a2 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:14:03.672625 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 723/1443] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:14:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:14:05.959079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb353de-a5d4-426b-b300-47e9c25930e7 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:05.959659 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb353de-a5d4-426b-b300-47e9c25930e7 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:05.959659 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb353de-a5d4-426b-b300-47e9c25930e7 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:05.960954 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 721/1444] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 04:14:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:14:05.966089 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8687ae4f-6682-491a-9bee-e8dbf7fe47b9 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:05.966541 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8687ae4f-6682-491a-9bee-e8dbf7fe47b9 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:05.966664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8687ae4f-6682-491a-9bee-e8dbf7fe47b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:06.031797 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-8687ae4f-6682-491a-9bee-e8dbf7fe47b9 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:14:06.033000 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-8687ae4f-6682-491a-9bee-e8dbf7fe47b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:06.045362 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:06.128877 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 724/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 04:14:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:14:06.411330 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb353de-a5d4-426b-b300-47e9c25930e7 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:14:06.412855 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-beb353de-a5d4-426b-b300-47e9c25930e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:06.513612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb353de-a5d4-426b-b300-47e9c25930e7 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:14:06.526454 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:06.586878 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3235b731-5c54-4a80-8872-d950f75ee6c9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:14:06.587477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3235b731-5c54-4a80-8872-d950f75ee6c9 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:06.587753 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3235b731-5c54-4a80-8872-d950f75ee6c9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:06.588025 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3235b731-5c54-4a80-8872-d950f75ee6c9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:06.589596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-3235b731-5c54-4a80-8872-d950f75ee6c9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:06.696120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3235b731-5c54-4a80-8872-d950f75ee6c9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:14:06.697454 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3235b731-5c54-4a80-8872-d950f75ee6c9 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:14:06.715420 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 722/1446] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:14:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:14:09.168443 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f05f07-1fa2-4b8e-bed1-313aa629a567 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:09.168974 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f05f07-1fa2-4b8e-bed1-313aa629a567 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:09.168974 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f05f07-1fa2-4b8e-bed1-313aa629a567 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:09.170290 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 725/1447] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 04:14:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:14:09.175643 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ed1831-1828-4320-a279-85e619d49a07 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:09.176053 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ed1831-1828-4320-a279-85e619d49a07 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:09.176196 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ed1831-1828-4320-a279-85e619d49a07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:09.224343 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:09.242603 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-a6ed1831-1828-4320-a279-85e619d49a07 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:14:09.243800 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a6ed1831-1828-4320-a279-85e619d49a07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:09.344659 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 723/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 04:14:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:14:09.635000 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f05f07-1fa2-4b8e-bed1-313aa629a567 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:14:09.636566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-53f05f07-1fa2-4b8e-bed1-313aa629a567 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:09.738626 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f05f07-1fa2-4b8e-bed1-313aa629a567 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:14:09.778189 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:09.811677 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b4ae88-89b7-4b76-aa6e-6630eff5c599 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:14:09.812275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b4ae88-89b7-4b76-aa6e-6630eff5c599 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:09.812551 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b4ae88-89b7-4b76-aa6e-6630eff5c599 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:09.812820 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b4ae88-89b7-4b76-aa6e-6630eff5c599 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:09.814417 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-e0b4ae88-89b7-4b76-aa6e-6630eff5c599 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:09.919569 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b4ae88-89b7-4b76-aa6e-6630eff5c599 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:14:09.920896 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b4ae88-89b7-4b76-aa6e-6630eff5c599 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:14:09.938062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 726/1449] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:14:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:14:12.205067 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775e02cb-2828-47b6-a5f4-e54eea3bd30b None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:12.205661 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775e02cb-2828-47b6-a5f4-e54eea3bd30b None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:12.205661 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775e02cb-2828-47b6-a5f4-e54eea3bd30b None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:12.206979 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 724/1450] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 04:14:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:14:12.212254 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcd557b-9dc6-4944-b46f-020ed83913a3 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:12.212666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcd557b-9dc6-4944-b46f-020ed83913a3 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:12.212803 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcd557b-9dc6-4944-b46f-020ed83913a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:12.277944 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-0dcd557b-9dc6-4944-b46f-020ed83913a3 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:14:12.279162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0dcd557b-9dc6-4944-b46f-020ed83913a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:12.299809 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:12.376647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 727/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 04:14:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:14:12.656707 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775e02cb-2828-47b6-a5f4-e54eea3bd30b None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:14:12.658320 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-775e02cb-2828-47b6-a5f4-e54eea3bd30b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:12.758569 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775e02cb-2828-47b6-a5f4-e54eea3bd30b None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:14:12.778548 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:12.831206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366561f9-b972-42df-8e20-758ad53147f6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:14:12.832268 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366561f9-b972-42df-8e20-758ad53147f6 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:12.832760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366561f9-b972-42df-8e20-758ad53147f6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:12.833119 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366561f9-b972-42df-8e20-758ad53147f6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:12.834776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-366561f9-b972-42df-8e20-758ad53147f6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:12.942848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366561f9-b972-42df-8e20-758ad53147f6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:14:12.944194 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366561f9-b972-42df-8e20-758ad53147f6 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:14:12.962321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 725/1452] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:14:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:14:15.268136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f86f7b-0f6a-455a-9433-6b946f6609ed None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:15.268663 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f86f7b-0f6a-455a-9433-6b946f6609ed None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:15.268663 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f86f7b-0f6a-455a-9433-6b946f6609ed None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:15.270078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 728/1453] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 04:14:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:14:15.275841 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaee2e09-e1fc-4b3e-b8a7-62c7d62b7ff3 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:15.276254 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaee2e09-e1fc-4b3e-b8a7-62c7d62b7ff3 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:15.276430 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaee2e09-e1fc-4b3e-b8a7-62c7d62b7ff3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:15.342021 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:15.344755 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-eaee2e09-e1fc-4b3e-b8a7-62c7d62b7ff3 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:14:15.346577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-eaee2e09-e1fc-4b3e-b8a7-62c7d62b7ff3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:15.441826 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 726/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 04:14:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:14:15.729541 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f86f7b-0f6a-455a-9433-6b946f6609ed None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:14:15.731098 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-33f86f7b-0f6a-455a-9433-6b946f6609ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:15.835274 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f86f7b-0f6a-455a-9433-6b946f6609ed None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:14:15.883659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:15.907328 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32c3896-960d-4945-b68e-dfddeb4cd04f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:14:15.907925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32c3896-960d-4945-b68e-dfddeb4cd04f service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:15.908199 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32c3896-960d-4945-b68e-dfddeb4cd04f service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:15.908466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32c3896-960d-4945-b68e-dfddeb4cd04f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:15.910055 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-a32c3896-960d-4945-b68e-dfddeb4cd04f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:16.015648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32c3896-960d-4945-b68e-dfddeb4cd04f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:14:16.017316 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32c3896-960d-4945-b68e-dfddeb4cd04f service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:14:16.034791 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 729/1455] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:14:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:14:18.234961 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cb7274-0923-4bc9-8396-fe9713828799 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:18.235525 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cb7274-0923-4bc9-8396-fe9713828799 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:18.235525 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cb7274-0923-4bc9-8396-fe9713828799 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:18.236823 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 727/1456] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 04:14:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:14:18.242159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4656eec5-ea61-4194-a47e-8ce059bcb91d None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:18.242587 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4656eec5-ea61-4194-a47e-8ce059bcb91d None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:18.242708 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4656eec5-ea61-4194-a47e-8ce059bcb91d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:18.306799 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-4656eec5-ea61-4194-a47e-8ce059bcb91d None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:14:18.307987 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-4656eec5-ea61-4194-a47e-8ce059bcb91d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:18.336742 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:18.403921 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 730/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 04:14:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:14:18.696258 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cb7274-0923-4bc9-8396-fe9713828799 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:14:18.697818 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-18cb7274-0923-4bc9-8396-fe9713828799 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:18.800104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cb7274-0923-4bc9-8396-fe9713828799 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:14:18.828303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:18.872518 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4752c6e2-879d-4332-96f4-52b449cd559e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:14:18.873113 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4752c6e2-879d-4332-96f4-52b449cd559e service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:18.873386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4752c6e2-879d-4332-96f4-52b449cd559e service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:18.873652 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4752c6e2-879d-4332-96f4-52b449cd559e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:18.875353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-4752c6e2-879d-4332-96f4-52b449cd559e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:18.982018 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4752c6e2-879d-4332-96f4-52b449cd559e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:14:18.983433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4752c6e2-879d-4332-96f4-52b449cd559e service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:14:19.001577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 728/1458] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:14:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:14:21.168521 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a77cc3-ab6b-4c43-979e-c750e3b564ff None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:21.169068 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a77cc3-ab6b-4c43-979e-c750e3b564ff None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:21.169068 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a77cc3-ab6b-4c43-979e-c750e3b564ff None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:21.170363 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 731/1459] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 04:14:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:14:21.175500 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ccfbcb-5b5a-4e51-9831-2a856ec9dc8f None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:21.175925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ccfbcb-5b5a-4e51-9831-2a856ec9dc8f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:21.176047 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ccfbcb-5b5a-4e51-9831-2a856ec9dc8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:21.239896 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-84ccfbcb-5b5a-4e51-9831-2a856ec9dc8f None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:14:21.241071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-84ccfbcb-5b5a-4e51-9831-2a856ec9dc8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:21.249258 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:21.334851 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 729/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 04:14:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:14:21.617079 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a77cc3-ab6b-4c43-979e-c750e3b564ff None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:14:21.618616 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-01a77cc3-ab6b-4c43-979e-c750e3b564ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:21.719551 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a77cc3-ab6b-4c43-979e-c750e3b564ff None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:14:21.773651 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:21.791731 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8161e0b8-bb99-4fd9-849c-3720375e5510 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:14:21.792324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8161e0b8-bb99-4fd9-849c-3720375e5510 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:21.792600 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8161e0b8-bb99-4fd9-849c-3720375e5510 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:21.792868 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8161e0b8-bb99-4fd9-849c-3720375e5510 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:21.794471 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-8161e0b8-bb99-4fd9-849c-3720375e5510 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:21.899294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8161e0b8-bb99-4fd9-849c-3720375e5510 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:14:21.900624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8161e0b8-bb99-4fd9-849c-3720375e5510 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:14:21.918470 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 732/1461] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:14:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:14:24.119342 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f92359-7ff0-4c08-938f-c3a882383702 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:24.119906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f92359-7ff0-4c08-938f-c3a882383702 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:24.119906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f92359-7ff0-4c08-938f-c3a882383702 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:24.121204 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 730/1462] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 04:14:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:14:24.126553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0519fc4-f4ed-49ed-b10e-5c901ea10b42 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:24.126984 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0519fc4-f4ed-49ed-b10e-5c901ea10b42 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:24.127107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0519fc4-f4ed-49ed-b10e-5c901ea10b42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:24.192011 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-e0519fc4-f4ed-49ed-b10e-5c901ea10b42 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:14:24.193216 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-e0519fc4-f4ed-49ed-b10e-5c901ea10b42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:24.230536 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:24.289187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 733/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 04:14:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:14:24.570436 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f92359-7ff0-4c08-938f-c3a882383702 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:14:24.571949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-96f92359-7ff0-4c08-938f-c3a882383702 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:24.673707 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f92359-7ff0-4c08-938f-c3a882383702 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:14:24.711381 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:24.747194 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fffd2e-e947-4792-afdc-6f7f61ece127 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:14:24.747793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fffd2e-e947-4792-afdc-6f7f61ece127 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:24.748067 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fffd2e-e947-4792-afdc-6f7f61ece127 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:24.748337 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fffd2e-e947-4792-afdc-6f7f61ece127 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:24.749926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d9fffd2e-e947-4792-afdc-6f7f61ece127 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:24.868548 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fffd2e-e947-4792-afdc-6f7f61ece127 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:14:24.870084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fffd2e-e947-4792-afdc-6f7f61ece127 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:14:24.887675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 731/1464] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:14:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:14:27.031491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af29d491-a683-41b6-89db-fcd2a29072a6 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:27.032049 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af29d491-a683-41b6-89db-fcd2a29072a6 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:27.032049 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af29d491-a683-41b6-89db-fcd2a29072a6 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:27.033433 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 734/1465] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 04:14:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:14:27.039738 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764c3228-c0f9-492d-a743-153f3f99d79f None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:27.040189 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764c3228-c0f9-492d-a743-153f3f99d79f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:27.040295 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764c3228-c0f9-492d-a743-153f3f99d79f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:27.104781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-764c3228-c0f9-492d-a743-153f3f99d79f None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:14:27.106088 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-764c3228-c0f9-492d-a743-153f3f99d79f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:27.124433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:27.200938 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 732/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 04:14:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:14:27.484293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af29d491-a683-41b6-89db-fcd2a29072a6 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:14:27.485841 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-af29d491-a683-41b6-89db-fcd2a29072a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:27.588537 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af29d491-a683-41b6-89db-fcd2a29072a6 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:14:27.654934 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:27.662321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8308dc-5f43-4f6b-9614-4e370ea8d889 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:14:27.662865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8308dc-5f43-4f6b-9614-4e370ea8d889 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:27.663146 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8308dc-5f43-4f6b-9614-4e370ea8d889 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:27.663421 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8308dc-5f43-4f6b-9614-4e370ea8d889 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:27.665012 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-6e8308dc-5f43-4f6b-9614-4e370ea8d889 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:27.772398 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8308dc-5f43-4f6b-9614-4e370ea8d889 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:14:27.773837 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8308dc-5f43-4f6b-9614-4e370ea8d889 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:14:27.791465 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 735/1467] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:14:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:14:29.975482 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2197f6-41b3-4fee-9f60-4fed034931aa None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:29.976645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2197f6-41b3-4fee-9f60-4fed034931aa None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:29.976645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2197f6-41b3-4fee-9f60-4fed034931aa None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:29.977949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 733/1468] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 04:14:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:14:29.983149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287a598f-b10b-4f4a-88b5-c9f371e1f330 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:29.983589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287a598f-b10b-4f4a-88b5-c9f371e1f330 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:29.983694 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287a598f-b10b-4f4a-88b5-c9f371e1f330 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:30.049321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-287a598f-b10b-4f4a-88b5-c9f371e1f330 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:14:30.050576 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-287a598f-b10b-4f4a-88b5-c9f371e1f330 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:30.098705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:30.148839 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 736/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 04:14:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:14:30.442033 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2197f6-41b3-4fee-9f60-4fed034931aa None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:14:30.443548 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-4b2197f6-41b3-4fee-9f60-4fed034931aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:30.545866 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2197f6-41b3-4fee-9f60-4fed034931aa None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:14:30.595094 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:30.620101 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff618e0-3d1f-44c6-95bb-f2e5b1d8c2f4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:14:30.620701 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff618e0-3d1f-44c6-95bb-f2e5b1d8c2f4 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:30.620975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff618e0-3d1f-44c6-95bb-f2e5b1d8c2f4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:30.621245 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff618e0-3d1f-44c6-95bb-f2e5b1d8c2f4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:30.622869 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-eff618e0-3d1f-44c6-95bb-f2e5b1d8c2f4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:30.729037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff618e0-3d1f-44c6-95bb-f2e5b1d8c2f4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:14:30.730484 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff618e0-3d1f-44c6-95bb-f2e5b1d8c2f4 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:14:30.748498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 734/1470] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:14:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:14:40.146722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=22 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:40.747301 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=66 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:50.713782 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e7b52f-ad3c-47ac-9f60-162b7be14bf9 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:50.714225 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e7b52f-ad3c-47ac-9f60-162b7be14bf9 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:50.714330 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e7b52f-ad3c-47ac-9f60-162b7be14bf9 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:50.715679 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 737/1471] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:14:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:14:50.720808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0c8c92-124e-4156-8198-3ab2cf8e100a None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:50.721258 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0c8c92-124e-4156-8198-3ab2cf8e100a None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:50.721366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0c8c92-124e-4156-8198-3ab2cf8e100a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:50.786365 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-0e0c8c92-124e-4156-8198-3ab2cf8e100a None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:14:50.787583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-0e0c8c92-124e-4156-8198-3ab2cf8e100a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:50.882123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 735/1472] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:14:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:14:51.181639 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d8c664-7679-4d9e-8e8e-cc91a5042e18 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:51.181948 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d8c664-7679-4d9e-8e8e-cc91a5042e18 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:51.182221 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d8c664-7679-4d9e-8e8e-cc91a5042e18 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:51.183555 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 738/1473] 10.222.0.26 () {54 vars in 962 bytes} [Tue Sep 27 04:14:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:14:51.188399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbf8d17-24d4-4ccc-95f5-5ef2e066051c None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:51.188685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbf8d17-24d4-4ccc-95f5-5ef2e066051c None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:51.188958 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbf8d17-24d4-4ccc-95f5-5ef2e066051c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:51.263171 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-6dbf8d17-24d4-4ccc-95f5-5ef2e066051c None None] MFA Rules not processed for user `8dcc53332e6b462baf349a72b19d2b3b`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:14:51.264349 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6dbf8d17-24d4-4ccc-95f5-5ef2e066051c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:51.284808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:51.338950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 736/1474] 10.222.0.26 () {58 vars in 1059 bytes} [Tue Sep 27 04:14:51 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:14:51.346657 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d8c664-7679-4d9e-8e8e-cc91a5042e18 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:14:51.348162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-37d8c664-7679-4d9e-8e8e-cc91a5042e18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:51.452840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d8c664-7679-4d9e-8e8e-cc91a5042e18 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:14:51.642453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14080c7f-8611-4803-bbba-868fbf302b60 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dcc53332e6b462baf349a72b19d2b3b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:14:51.643051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14080c7f-8611-4803-bbba-868fbf302b60 service glance] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:51.643329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14080c7f-8611-4803-bbba-868fbf302b60 service glance] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:51.643597 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14080c7f-8611-4803-bbba-868fbf302b60 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:51.645178 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-14080c7f-8611-4803-bbba-868fbf302b60 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:51.699120 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:51.755513 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14080c7f-8611-4803-bbba-868fbf302b60 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:14:51.757567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14080c7f-8611-4803-bbba-868fbf302b60 service glance] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:14:51.775367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 739/1475] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:14:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:14:53.981830 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b5b848-b4ae-4e8b-8e35-8c34a2f2c19d None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:53.982346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b5b848-b4ae-4e8b-8e35-8c34a2f2c19d None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:53.982462 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b5b848-b4ae-4e8b-8e35-8c34a2f2c19d None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:53.983695 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 737/1476] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:14:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:14:53.988854 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea71aec-1487-4cdc-b005-87f778818fae None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:53.989280 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea71aec-1487-4cdc-b005-87f778818fae None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:53.989376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea71aec-1487-4cdc-b005-87f778818fae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:54.053501 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-4ea71aec-1487-4cdc-b005-87f778818fae None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:14:54.054740 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-4ea71aec-1487-4cdc-b005-87f778818fae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:54.150736 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 740/1477] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:14:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:14:54.433041 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b5b848-b4ae-4e8b-8e35-8c34a2f2c19d None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:14:54.434623 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-82b5b848-b4ae-4e8b-8e35-8c34a2f2c19d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:54.557835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b5b848-b4ae-4e8b-8e35-8c34a2f2c19d None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:14:54.606783 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:54.650496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0007dca7-ecf9-4857-a302-5ce4d6114b5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:14:54.651098 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0007dca7-ecf9-4857-a302-5ce4d6114b5f admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:54.651373 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0007dca7-ecf9-4857-a302-5ce4d6114b5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:54.651642 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0007dca7-ecf9-4857-a302-5ce4d6114b5f admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:54.652270 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0007dca7-ecf9-4857-a302-5ce4d6114b5f admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:14:54.654063 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: WARNING py.warnings [None req-0007dca7-ecf9-4857-a302-5ce4d6114b5f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 27 04:14:54.654063 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: warnings.warn(msg) Sep 27 04:14:54.654063 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: Sep 27 04:14:54.655190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0007dca7-ecf9-4857-a302-5ce4d6114b5f admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:14:54.674061 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 738/1478] 10.222.0.26 () {56 vars in 1108 bytes} [Tue Sep 27 04:14:54 2022] GET /identity/v3/projects => generated 2553 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:14:56.850361 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fed9e2-b89b-4586-98bf-89639ac3ce5b None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:56.851391 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fed9e2-b89b-4586-98bf-89639ac3ce5b None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:56.851508 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fed9e2-b89b-4586-98bf-89639ac3ce5b None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:56.852769 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 741/1479] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:14:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:14:56.858024 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39eeb4f4-2d6e-4c3f-bfce-709fc1b958f2 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:56.858424 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39eeb4f4-2d6e-4c3f-bfce-709fc1b958f2 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:56.858613 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39eeb4f4-2d6e-4c3f-bfce-709fc1b958f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:56.925009 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-39eeb4f4-2d6e-4c3f-bfce-709fc1b958f2 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:14:56.926251 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-39eeb4f4-2d6e-4c3f-bfce-709fc1b958f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:57.023151 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 739/1480] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:14:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:14:57.317269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fed9e2-b89b-4586-98bf-89639ac3ce5b None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:14:57.318793 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-f9fed9e2-b89b-4586-98bf-89639ac3ce5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:57.329339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:57.465655 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fed9e2-b89b-4586-98bf-89639ac3ce5b None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:14:57.544258 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:14:57.576974 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60be41b-0578-439e-a9e8-eb062380638b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:14:57.577510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60be41b-0578-439e-a9e8-eb062380638b service nova] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:14:57.577823 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60be41b-0578-439e-a9e8-eb062380638b service nova] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:14:57.578100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60be41b-0578-439e-a9e8-eb062380638b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:14:57.579675 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-a60be41b-0578-439e-a9e8-eb062380638b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:14:57.667345 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60be41b-0578-439e-a9e8-eb062380638b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:14:57.668950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60be41b-0578-439e-a9e8-eb062380638b service nova] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:14:57.686363 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 742/1481] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:14:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:15:00.035539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78d89d9-44c8-4d56-b651-f47b5e79ca5f None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:15:00.036108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78d89d9-44c8-4d56-b651-f47b5e79ca5f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:15:00.036108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78d89d9-44c8-4d56-b651-f47b5e79ca5f None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:15:00.037430 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 740/1482] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:15:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:15:00.042900 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06680efb-14ec-4267-827a-4cddbb29848a None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:15:00.043347 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06680efb-14ec-4267-827a-4cddbb29848a None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:15:00.043464 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06680efb-14ec-4267-827a-4cddbb29848a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:15:00.108479 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-06680efb-14ec-4267-827a-4cddbb29848a None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:15:00.109687 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-06680efb-14ec-4267-827a-4cddbb29848a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:15:00.156111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:15:00.207039 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 743/1483] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:15:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:15:00.489837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78d89d9-44c8-4d56-b651-f47b5e79ca5f None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:15:00.492107 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-f78d89d9-44c8-4d56-b651-f47b5e79ca5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:15:00.515336 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:15:00.593999 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78d89d9-44c8-4d56-b651-f47b5e79ca5f None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:15:00.784762 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19c446f-d087-4b0c-8813-277b8e1e3bf7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:15:00.785361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19c446f-d087-4b0c-8813-277b8e1e3bf7 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:15:00.785635 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19c446f-d087-4b0c-8813-277b8e1e3bf7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:15:00.785932 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19c446f-d087-4b0c-8813-277b8e1e3bf7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:15:00.787508 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d19c446f-d087-4b0c-8813-277b8e1e3bf7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:15:00.859903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:15:00.895516 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19c446f-d087-4b0c-8813-277b8e1e3bf7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:15:00.896948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19c446f-d087-4b0c-8813-277b8e1e3bf7 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:15:00.915626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 741/1484] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:15:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:15:03.138388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27bbe41-4d84-42a1-bb1f-25ab50217003 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:15:03.138923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27bbe41-4d84-42a1-bb1f-25ab50217003 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:15:03.138923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27bbe41-4d84-42a1-bb1f-25ab50217003 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:15:03.140207 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 744/1485] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:15:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:15:03.145580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3c89d6-d26c-4216-a944-0b41915d1206 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:15:03.146058 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3c89d6-d26c-4216-a944-0b41915d1206 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:15:03.146173 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3c89d6-d26c-4216-a944-0b41915d1206 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:15:03.210010 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-ee3c89d6-d26c-4216-a944-0b41915d1206 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:15:03.211200 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ee3c89d6-d26c-4216-a944-0b41915d1206 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:15:03.304824 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 742/1486] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:15:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:15:03.595782 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27bbe41-4d84-42a1-bb1f-25ab50217003 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:15:03.597294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-d27bbe41-4d84-42a1-bb1f-25ab50217003 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:15:03.699223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27bbe41-4d84-42a1-bb1f-25ab50217003 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:15:03.769465 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:15:03.771226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29dd193-0247-48da-8c72-05ba51b3e72d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:15:03.771916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29dd193-0247-48da-8c72-05ba51b3e72d service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:15:03.772353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29dd193-0247-48da-8c72-05ba51b3e72d service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:15:03.773388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29dd193-0247-48da-8c72-05ba51b3e72d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:15:03.775399 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-e29dd193-0247-48da-8c72-05ba51b3e72d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:15:03.879808 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29dd193-0247-48da-8c72-05ba51b3e72d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:15:03.881204 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29dd193-0247-48da-8c72-05ba51b3e72d service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:15:03.899119 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 745/1487] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:15:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:15:06.173483 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a26928-e6d5-4016-b73d-ff049a47c6eb None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:15:06.174108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a26928-e6d5-4016-b73d-ff049a47c6eb None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:15:06.174108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a26928-e6d5-4016-b73d-ff049a47c6eb None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:15:06.175394 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 743/1488] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:15:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:15:06.181368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0454e71b-5e86-4190-91ac-becd3c538e80 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:15:06.181827 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0454e71b-5e86-4190-91ac-becd3c538e80 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:15:06.181930 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0454e71b-5e86-4190-91ac-becd3c538e80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:15:06.246437 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-0454e71b-5e86-4190-91ac-becd3c538e80 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:15:06.247637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0454e71b-5e86-4190-91ac-becd3c538e80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:15:06.299069 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:15:06.342870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 746/1489] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:15:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:15:06.636285 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a26928-e6d5-4016-b73d-ff049a47c6eb None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:15:06.637825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-80a26928-e6d5-4016-b73d-ff049a47c6eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:15:06.673470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:15:06.741011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a26928-e6d5-4016-b73d-ff049a47c6eb None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:15:06.816027 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0a4fe5-eb41-47d3-bf70-0b4e615f0375 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:15:06.816627 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0a4fe5-eb41-47d3-bf70-0b4e615f0375 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:15:06.816902 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0a4fe5-eb41-47d3-bf70-0b4e615f0375 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:15:06.817175 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0a4fe5-eb41-47d3-bf70-0b4e615f0375 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:15:06.818809 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-1e0a4fe5-eb41-47d3-bf70-0b4e615f0375 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:15:06.895810 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:15:06.925851 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0a4fe5-eb41-47d3-bf70-0b4e615f0375 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:15:06.927403 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0a4fe5-eb41-47d3-bf70-0b4e615f0375 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:15:06.945391 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 744/1490] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:15:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:15:09.374210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79346086-2841-42cd-b274-b39e14327d1c None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:15:09.374771 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79346086-2841-42cd-b274-b39e14327d1c None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:15:09.374771 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79346086-2841-42cd-b274-b39e14327d1c None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:15:09.376071 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 747/1491] 10.222.0.26 () {54 vars in 927 bytes} [Tue Sep 27 04:15:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:15:09.381241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39f501d-5158-4ab1-8897-0a7cd6cd4cb5 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:15:09.381681 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39f501d-5158-4ab1-8897-0a7cd6cd4cb5 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:15:09.381822 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39f501d-5158-4ab1-8897-0a7cd6cd4cb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:15:09.447691 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-b39f501d-5158-4ab1-8897-0a7cd6cd4cb5 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:15:09.448873 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-b39f501d-5158-4ab1-8897-0a7cd6cd4cb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:15:09.556302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 745/1492] 10.222.0.26 () {58 vars in 1024 bytes} [Tue Sep 27 04:15:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:15:09.844459 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79346086-2841-42cd-b274-b39e14327d1c None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:15:09.846012 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-79346086-2841-42cd-b274-b39e14327d1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:15:09.947022 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79346086-2841-42cd-b274-b39e14327d1c None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:15:10.018517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e83b6b54-618e-4a03-92a6-1c4548ac0b7f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:15:10.019111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83b6b54-618e-4a03-92a6-1c4548ac0b7f service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:15:10.019386 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83b6b54-618e-4a03-92a6-1c4548ac0b7f service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:15:10.019657 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83b6b54-618e-4a03-92a6-1c4548ac0b7f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:15:10.021223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-e83b6b54-618e-4a03-92a6-1c4548ac0b7f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:15:10.030076 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:15:10.127406 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e83b6b54-618e-4a03-92a6-1c4548ac0b7f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:15:10.128741 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e83b6b54-618e-4a03-92a6-1c4548ac0b7f service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:15:10.146880 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 748/1493] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:15:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:15:19.554302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=82 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:15:20.144664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=50 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:16:27.874565 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb79dc56-810c-4d1a-8fca-8a5571936e16 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:16:27.875152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb79dc56-810c-4d1a-8fca-8a5571936e16 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:16:27.875152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb79dc56-810c-4d1a-8fca-8a5571936e16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:16:27.948608 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-fb79dc56-810c-4d1a-8fca-8a5571936e16 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:16:27.949809 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-fb79dc56-810c-4d1a-8fca-8a5571936e16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:16:28.041674 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 746/1494] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:16:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:16:28.049449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1690bbb0-165e-40f8-bea4-5fd710be4fa8 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:16:28.049972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1690bbb0-165e-40f8-bea4-5fd710be4fa8 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:16:28.050091 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1690bbb0-165e-40f8-bea4-5fd710be4fa8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:16:28.124378 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-1690bbb0-165e-40f8-bea4-5fd710be4fa8 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:16:28.125619 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-1690bbb0-165e-40f8-bea4-5fd710be4fa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:16:28.220162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 749/1495] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:16:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:16:28.224747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb79dc56-810c-4d1a-8fca-8a5571936e16 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:16:28.225947 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-fb79dc56-810c-4d1a-8fca-8a5571936e16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:16:28.237366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb79dc56-810c-4d1a-8fca-8a5571936e16 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:16:28.296035 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:16:28.328102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07d2f1e-b5a8-4de2-bb07-45da089954e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:16:28.328694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07d2f1e-b5a8-4de2-bb07-45da089954e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:16:28.328963 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07d2f1e-b5a8-4de2-bb07-45da089954e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:16:28.329225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07d2f1e-b5a8-4de2-bb07-45da089954e6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:16:28.329888 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07d2f1e-b5a8-4de2-bb07-45da089954e6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:16:28.330181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07d2f1e-b5a8-4de2-bb07-45da089954e6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:16:28.331171 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: WARNING py.warnings [None req-d07d2f1e-b5a8-4de2-bb07-45da089954e6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 27 04:16:28.331171 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: warnings.warn(msg) Sep 27 04:16:28.331171 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: Sep 27 04:16:28.332035 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07d2f1e-b5a8-4de2-bb07-45da089954e6 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:16:28.342771 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 747/1496] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:16:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:16:28.350996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1690bbb0-165e-40f8-bea4-5fd710be4fa8 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:16:28.360504 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1690bbb0-165e-40f8-bea4-5fd710be4fa8 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:16:28.421230 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:16:28.454264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7f2219-1d6f-4e74-8a9d-d95b6fa89e09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:16:28.454872 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7f2219-1d6f-4e74-8a9d-d95b6fa89e09 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:16:28.455151 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7f2219-1d6f-4e74-8a9d-d95b6fa89e09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:16:28.455424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7f2219-1d6f-4e74-8a9d-d95b6fa89e09 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:16:28.456060 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7f2219-1d6f-4e74-8a9d-d95b6fa89e09 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:16:28.457820 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: WARNING py.warnings [None req-ae7f2219-1d6f-4e74-8a9d-d95b6fa89e09 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 27 04:16:28.457820 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: warnings.warn(msg) Sep 27 04:16:28.457820 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: Sep 27 04:16:28.459006 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7f2219-1d6f-4e74-8a9d-d95b6fa89e09 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:16:28.515322 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 750/1497] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:16:28 2022] POST /identity/v3/projects => generated 377 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:16:28.519541 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07d2f1e-b5a8-4de2-bb07-45da089954e6 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:16:28.521354 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d07d2f1e-b5a8-4de2-bb07-45da089954e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:16:28.554653 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07d2f1e-b5a8-4de2-bb07-45da089954e6 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:16:28.644439 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2023898e-2b31-4e1a-95df-f5ea2ef994e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:16:28.645032 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2023898e-2b31-4e1a-95df-f5ea2ef994e2 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:16:28.645559 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2023898e-2b31-4e1a-95df-f5ea2ef994e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:16:28.645869 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2023898e-2b31-4e1a-95df-f5ea2ef994e2 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:16:28.646498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2023898e-2b31-4e1a-95df-f5ea2ef994e2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:16:28.647496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: WARNING py.warnings [None req-2023898e-2b31-4e1a-95df-f5ea2ef994e2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 27 04:16:28.647496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: warnings.warn(msg) Sep 27 04:16:28.647496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: Sep 27 04:16:28.649168 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2023898e-2b31-4e1a-95df-f5ea2ef994e2 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:16:28.689611 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 748/1498] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:16:28 2022] POST /identity/v3/users => generated 289 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:16:28.693758 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7f2219-1d6f-4e74-8a9d-d95b6fa89e09 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:16:28.695307 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ae7f2219-1d6f-4e74-8a9d-d95b6fa89e09 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:16:28.755752 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7f2219-1d6f-4e74-8a9d-d95b6fa89e09 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:16:28.838400 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:16:28.849302 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2b496b-3b04-4a1c-8182-b6ff8626c9d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:16:28.849933 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2b496b-3b04-4a1c-8182-b6ff8626c9d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:16:28.850208 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2b496b-3b04-4a1c-8182-b6ff8626c9d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:16:28.850498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2b496b-3b04-4a1c-8182-b6ff8626c9d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:16:28.851132 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2b496b-3b04-4a1c-8182-b6ff8626c9d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:16:28.852120 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: WARNING py.warnings [None req-2e2b496b-3b04-4a1c-8182-b6ff8626c9d4 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 27 04:16:28.852120 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: warnings.warn(msg) Sep 27 04:16:28.852120 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: Sep 27 04:16:28.852992 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2b496b-3b04-4a1c-8182-b6ff8626c9d4 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:16:28.864914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 751/1499] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:16:28 2022] GET /identity/v3/roles => generated 1420 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:16:28.868935 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2023898e-2b31-4e1a-95df-f5ea2ef994e2 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:16:28.870527 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-2023898e-2b31-4e1a-95df-f5ea2ef994e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:16:28.881608 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2023898e-2b31-4e1a-95df-f5ea2ef994e2 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:16:28.907086 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:16:28.975306 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1354ff8f-d91c-441e-bb98-7bfc6bdb45e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:16:28.975905 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1354ff8f-d91c-441e-bb98-7bfc6bdb45e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:16:28.976172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1354ff8f-d91c-441e-bb98-7bfc6bdb45e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:16:28.976433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1354ff8f-d91c-441e-bb98-7bfc6bdb45e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:16:28.977592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1354ff8f-d91c-441e-bb98-7bfc6bdb45e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:16:28.978623 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: WARNING py.warnings [None req-1354ff8f-d91c-441e-bb98-7bfc6bdb45e8 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 27 04:16:28.978623 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: warnings.warn(msg) Sep 27 04:16:28.978623 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: Sep 27 04:16:28.979497 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1354ff8f-d91c-441e-bb98-7bfc6bdb45e8 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:16:28.991322 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 749/1500] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:16:28 2022] GET /identity/v3/roles => generated 1420 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:16:28.996068 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2b496b-3b04-4a1c-8182-b6ff8626c9d4 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:16:28.997553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-2e2b496b-3b04-4a1c-8182-b6ff8626c9d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:16:29.029760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2b496b-3b04-4a1c-8182-b6ff8626c9d4 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:16:29.121768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4b4109-9cfa-4a1e-894b-b984d1b33c7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:16:29.122388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4b4109-9cfa-4a1e-894b-b984d1b33c7b admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:16:29.122630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4b4109-9cfa-4a1e-894b-b984d1b33c7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:16:29.122901 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4b4109-9cfa-4a1e-894b-b984d1b33c7b admin admin] PATH_INFO: `/v3/projects/83f865ed3f584f7b9b21f6efd33e6825/users/a8302cbd880e49d1ab2cf3192b56de1c/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:16:29.144930 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4b4109-9cfa-4a1e-894b-b984d1b33c7b admin admin] RBAC: Authorizing `identity:create_grant(project_id=83f865ed3f584f7b9b21f6efd33e6825, user_id=a8302cbd880e49d1ab2cf3192b56de1c, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:16:29.146243 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: WARNING py.warnings [None req-8c4b4109-9cfa-4a1e-894b-b984d1b33c7b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 27 04:16:29.146243 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: warnings.warn(msg) Sep 27 04:16:29.146243 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: Sep 27 04:16:29.148158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4b4109-9cfa-4a1e-894b-b984d1b33c7b admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:16:29.172493 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 752/1501] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:16:28 2022] PUT /identity/v3/projects/83f865ed3f584f7b9b21f6efd33e6825/users/a8302cbd880e49d1ab2cf3192b56de1c/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:16:29.184528 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013efcd9-8414-492a-8e50-cba44748a6c5 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:16:29.184809 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013efcd9-8414-492a-8e50-cba44748a6c5 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:16:29.185089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013efcd9-8414-492a-8e50-cba44748a6c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:16:29.239589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-013efcd9-8414-492a-8e50-cba44748a6c5 None None] MFA Rules not processed for user `a8302cbd880e49d1ab2cf3192b56de1c`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:16:29.240766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-013efcd9-8414-492a-8e50-cba44748a6c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:16:29.315478 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 750/1502] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:16:29 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:16:29.328587 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07d0063-d7e8-4c22-963d-d540605794d6 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:16:29.328877 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07d0063-d7e8-4c22-963d-d540605794d6 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:16:29.329150 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07d0063-d7e8-4c22-963d-d540605794d6 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:16:29.330488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 753/1503] 10.222.0.26 () {54 vars in 963 bytes} [Tue Sep 27 04:16:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:16:29.335335 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8012081b-334a-415d-b4b3-00fad870e5c9 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:16:29.335620 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8012081b-334a-415d-b4b3-00fad870e5c9 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:16:29.335896 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8012081b-334a-415d-b4b3-00fad870e5c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:16:29.367299 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:16:29.413413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-8012081b-334a-415d-b4b3-00fad870e5c9 None None] MFA Rules not processed for user `3970310fc7e34704b686e8d3bff3fdb2`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:16:29.415288 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-8012081b-334a-415d-b4b3-00fad870e5c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:16:29.489429 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 751/1504] 10.222.0.26 () {58 vars in 1060 bytes} [Tue Sep 27 04:16:29 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:16:29.497142 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07d0063-d7e8-4c22-963d-d540605794d6 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:16:29.498699 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-e07d0063-d7e8-4c22-963d-d540605794d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:16:29.547610 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:16:29.601094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07d0063-d7e8-4c22-963d-d540605794d6 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:16:29.673904 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf726da-e823-4530-9fd3-647e81d293bf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:16:29.674338 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf726da-e823-4530-9fd3-647e81d293bf service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:16:29.674612 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf726da-e823-4530-9fd3-647e81d293bf service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:16:29.674878 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf726da-e823-4530-9fd3-647e81d293bf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:16:29.676427 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-fcf726da-e823-4530-9fd3-647e81d293bf service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:16:29.771116 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf726da-e823-4530-9fd3-647e81d293bf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:16:29.772521 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf726da-e823-4530-9fd3-647e81d293bf service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:16:29.776570 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:16:29.790949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 754/1505] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:16:29 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:16:29.836385 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdf255f-eeb0-4929-a537-3fd51a75aa78 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:16:29.837376 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdf255f-eeb0-4929-a537-3fd51a75aa78 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:16:29.837912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdf255f-eeb0-4929-a537-3fd51a75aa78 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:16:29.839873 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 752/1506] 10.222.0.26 () {52 vars in 830 bytes} [Tue Sep 27 04:16:29 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:16:29.843715 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf726da-e823-4530-9fd3-647e81d293bf service cinder] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:16:29.844859 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-fcf726da-e823-4530-9fd3-647e81d293bf service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:16:29.941768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf726da-e823-4530-9fd3-647e81d293bf service cinder] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:16:30.033231 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c40977b-f328-4fbf-b0c3-bcced1bb5e5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:16:30.033876 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c40977b-f328-4fbf-b0c3-bcced1bb5e5f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:16:30.034149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c40977b-f328-4fbf-b0c3-bcced1bb5e5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:16:30.034412 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c40977b-f328-4fbf-b0c3-bcced1bb5e5f admin admin] PATH_INFO: `/v3/users/a8302cbd880e49d1ab2cf3192b56de1c` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:16:30.046856 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c40977b-f328-4fbf-b0c3-bcced1bb5e5f admin admin] RBAC: Authorizing `identity:delete_user(user_id=a8302cbd880e49d1ab2cf3192b56de1c)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:16:30.047883 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: WARNING py.warnings [None req-0c40977b-f328-4fbf-b0c3-bcced1bb5e5f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 27 04:16:30.047883 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: warnings.warn(msg) Sep 27 04:16:30.047883 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: Sep 27 04:16:30.049067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c40977b-f328-4fbf-b0c3-bcced1bb5e5f admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:16:30.058897 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:16:30.116005 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0c40977b-f328-4fbf-b0c3-bcced1bb5e5f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a8302cbd880e49d1ab2cf3192b56de1c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:16:30.130196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0c40977b-f328-4fbf-b0c3-bcced1bb5e5f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a8302cbd880e49d1ab2cf3192b56de1c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:16:30.130785 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0c40977b-f328-4fbf-b0c3-bcced1bb5e5f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a8302cbd880e49d1ab2cf3192b56de1c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:16:30.140148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0c40977b-f328-4fbf-b0c3-bcced1bb5e5f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a8302cbd880e49d1ab2cf3192b56de1c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:16:30.163268 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0c40977b-f328-4fbf-b0c3-bcced1bb5e5f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a8302cbd880e49d1ab2cf3192b56de1c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:16:30.168792 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 755/1507] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:16:29 2022] DELETE /identity/v3/users/a8302cbd880e49d1ab2cf3192b56de1c => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:16:30.173261 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db4744b-325c-43e9-8273-d164132c36f3 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:16:30.173725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db4744b-325c-43e9-8273-d164132c36f3 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:16:30.173868 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db4744b-325c-43e9-8273-d164132c36f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:16:30.230687 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-8db4744b-325c-43e9-8273-d164132c36f3 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:16:30.231885 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-8db4744b-325c-43e9-8273-d164132c36f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:16:30.307049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:16:30.319648 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 753/1508] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:16:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:16:30.329520 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c40977b-f328-4fbf-b0c3-bcced1bb5e5f admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:16:30.330768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0c40977b-f328-4fbf-b0c3-bcced1bb5e5f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:16:30.432334 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c40977b-f328-4fbf-b0c3-bcced1bb5e5f admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:16:30.484849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:16:30.504904 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de0befb-bb3f-43c9-b46e-aa20b77afb23 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:16:30.505505 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de0befb-bb3f-43c9-b46e-aa20b77afb23 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:16:30.505802 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de0befb-bb3f-43c9-b46e-aa20b77afb23 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:16:30.506073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de0befb-bb3f-43c9-b46e-aa20b77afb23 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:16:30.507652 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-9de0befb-bb3f-43c9-b46e-aa20b77afb23 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:16:30.612909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de0befb-bb3f-43c9-b46e-aa20b77afb23 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:16:30.614378 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de0befb-bb3f-43c9-b46e-aa20b77afb23 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:16:30.633163 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 756/1509] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:16:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:16:30.971601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db4744b-325c-43e9-8273-d164132c36f3 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:16:30.972915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-8db4744b-325c-43e9-8273-d164132c36f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:16:30.985132 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db4744b-325c-43e9-8273-d164132c36f3 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:16:31.087023 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec55e85-7540-4356-bfb8-7fdfde015eec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:16:31.087647 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec55e85-7540-4356-bfb8-7fdfde015eec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:16:31.087923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec55e85-7540-4356-bfb8-7fdfde015eec admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:16:31.088191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec55e85-7540-4356-bfb8-7fdfde015eec admin admin] PATH_INFO: `/v3/projects/83f865ed3f584f7b9b21f6efd33e6825` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:16:31.097791 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec55e85-7540-4356-bfb8-7fdfde015eec admin admin] RBAC: Authorizing `identity:delete_project(project_id=83f865ed3f584f7b9b21f6efd33e6825)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:16:31.098802 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: WARNING py.warnings [None req-eec55e85-7540-4356-bfb8-7fdfde015eec admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 27 04:16:31.098802 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: warnings.warn(msg) Sep 27 04:16:31.098802 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: Sep 27 04:16:31.099982 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec55e85-7540-4356-bfb8-7fdfde015eec admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:16:31.157974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-eec55e85-7540-4356-bfb8-7fdfde015eec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '83f865ed3f584f7b9b21f6efd33e6825', 'request_id': 'req-eec55e85-7540-4356-bfb8-7fdfde015eec'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:16:31.163946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-eec55e85-7540-4356-bfb8-7fdfde015eec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '83f865ed3f584f7b9b21f6efd33e6825', 'request_id': 'req-eec55e85-7540-4356-bfb8-7fdfde015eec'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:16:31.177001 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-eec55e85-7540-4356-bfb8-7fdfde015eec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '83f865ed3f584f7b9b21f6efd33e6825', 'request_id': 'req-eec55e85-7540-4356-bfb8-7fdfde015eec'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:16:31.203052 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-eec55e85-7540-4356-bfb8-7fdfde015eec admin admin] The token cache is being invalidate because project 83f865ed3f584f7b9b21f6efd33e6825 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:16:31.203341 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-eec55e85-7540-4356-bfb8-7fdfde015eec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:16:31.203767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-eec55e85-7540-4356-bfb8-7fdfde015eec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:16:31.205131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 754/1510] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:16:30 2022] DELETE /identity/v3/projects/83f865ed3f584f7b9b21f6efd33e6825 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:16:40.630594 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=62 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:16:41.201069 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:17:59.351689 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85478ec2-c263-4c76-addc-ccb6763dd624 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:17:59.352313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85478ec2-c263-4c76-addc-ccb6763dd624 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:17:59.352313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85478ec2-c263-4c76-addc-ccb6763dd624 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:17:59.353562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 757/1511] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 27 04:17:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:17:59.358863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dc8de3-bb6e-4483-9626-373ec6ac1c91 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:17:59.359299 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dc8de3-bb6e-4483-9626-373ec6ac1c91 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:17:59.359410 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dc8de3-bb6e-4483-9626-373ec6ac1c91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:17:59.439492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-57dc8de3-bb6e-4483-9626-373ec6ac1c91 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:17:59.440685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-57dc8de3-bb6e-4483-9626-373ec6ac1c91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:17:59.553159 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 755/1512] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 27 04:17:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:17:59.704682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85478ec2-c263-4c76-addc-ccb6763dd624 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:17:59.706761 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-85478ec2-c263-4c76-addc-ccb6763dd624 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:17:59.823458 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85478ec2-c263-4c76-addc-ccb6763dd624 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:17:59.829434 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cede29-5858-440c-bcc8-63144e986d44 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:17:59.830063 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cede29-5858-440c-bcc8-63144e986d44 service nova] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:17:59.830348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cede29-5858-440c-bcc8-63144e986d44 service nova] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:17:59.830619 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cede29-5858-440c-bcc8-63144e986d44 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:17:59.838537 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cede29-5858-440c-bcc8-63144e986d44 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:17:59.840103 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cede29-5858-440c-bcc8-63144e986d44 service nova] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:17:59.850062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 758/1513] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:17:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:18:00.941716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483033e6-0d3f-41a3-b504-4519340adb17 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:00.942286 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483033e6-0d3f-41a3-b504-4519340adb17 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:00.942286 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483033e6-0d3f-41a3-b504-4519340adb17 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:00.943610 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 756/1514] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 27 04:18:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:18:00.950253 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7448918e-ee16-4329-8981-cb275a4c1efe None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:00.950708 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7448918e-ee16-4329-8981-cb275a4c1efe None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:00.950827 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7448918e-ee16-4329-8981-cb275a4c1efe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:00.970545 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-7448918e-ee16-4329-8981-cb275a4c1efe None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:18:00.971726 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-7448918e-ee16-4329-8981-cb275a4c1efe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:18:00.979706 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 759/1515] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 27 04:18:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:18:01.139195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483033e6-0d3f-41a3-b504-4519340adb17 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:18:01.157023 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483033e6-0d3f-41a3-b504-4519340adb17 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:18:01.162797 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8125cf88-2c60-49ec-bd4e-6d7f06f0481c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:18:01.163397 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8125cf88-2c60-49ec-bd4e-6d7f06f0481c service nova] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:01.163674 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8125cf88-2c60-49ec-bd4e-6d7f06f0481c service nova] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:01.163944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8125cf88-2c60-49ec-bd4e-6d7f06f0481c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:01.172764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8125cf88-2c60-49ec-bd4e-6d7f06f0481c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:18:01.174385 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8125cf88-2c60-49ec-bd4e-6d7f06f0481c service nova] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:18:01.183666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 757/1516] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:18:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:18:01.695703 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd5f794-00f4-4b0c-9691-71c97ad76b12 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:01.695927 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd5f794-00f4-4b0c-9691-71c97ad76b12 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:01.696181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd5f794-00f4-4b0c-9691-71c97ad76b12 None None] PATH_INFO: `/v3` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:01.697441 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 760/1517] 10.222.0.57 () {54 vars in 925 bytes} [Tue Sep 27 04:18:01 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:18:01.706676 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79e096a-6fe1-4448-b00b-36c1e3d55456 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:01.706963 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79e096a-6fe1-4448-b00b-36c1e3d55456 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:01.707239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79e096a-6fe1-4448-b00b-36c1e3d55456 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:01.766282 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-e79e096a-6fe1-4448-b00b-36c1e3d55456 None None] MFA Rules not processed for user `3a1bc198dfc0486e9e56d4fb28cda05d`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:18:01.767476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-e79e096a-6fe1-4448-b00b-36c1e3d55456 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:18:01.783989 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:18:01.840702 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 758/1518] 10.222.0.57 () {58 vars in 1015 bytes} [Tue Sep 27 04:18:01 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:18:05.496720 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807b536a-d60b-47c5-87f8-23bcfcb240e5 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:05.497285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807b536a-d60b-47c5-87f8-23bcfcb240e5 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:05.497285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807b536a-d60b-47c5-87f8-23bcfcb240e5 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:05.498663 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 761/1519] 10.222.0.26 () {54 vars in 961 bytes} [Tue Sep 27 04:18:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:18:05.504047 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fa1ddf-84d4-43a0-8b93-becb67a5ba66 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:05.504528 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fa1ddf-84d4-43a0-8b93-becb67a5ba66 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:05.504632 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fa1ddf-84d4-43a0-8b93-becb67a5ba66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:05.524384 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-c0fa1ddf-84d4-43a0-8b93-becb67a5ba66 None None] MFA Rules not processed for user `3a1bc198dfc0486e9e56d4fb28cda05d`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:18:05.526262 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c0fa1ddf-84d4-43a0-8b93-becb67a5ba66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:18:05.536516 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 759/1520] 10.222.0.26 () {58 vars in 1058 bytes} [Tue Sep 27 04:18:05 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:18:05.544399 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807b536a-d60b-47c5-87f8-23bcfcb240e5 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:18:05.558224 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807b536a-d60b-47c5-87f8-23bcfcb240e5 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:18:05.563117 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c75ce4-d7dc-4381-b887-34efb9a7ac95 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3a1bc198dfc0486e9e56d4fb28cda05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:18:05.563702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c75ce4-d7dc-4381-b887-34efb9a7ac95 service placement] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:05.563982 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c75ce4-d7dc-4381-b887-34efb9a7ac95 service placement] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:05.564255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c75ce4-d7dc-4381-b887-34efb9a7ac95 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:05.572248 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c75ce4-d7dc-4381-b887-34efb9a7ac95 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:18:05.573818 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c75ce4-d7dc-4381-b887-34efb9a7ac95 service placement] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:18:05.583112 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 762/1521] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:18:05 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:18:12.653617 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10208173-0602-4c87-8cb7-6eff0252ef85 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:12.654271 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10208173-0602-4c87-8cb7-6eff0252ef85 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:12.654271 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10208173-0602-4c87-8cb7-6eff0252ef85 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:12.655536 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 760/1522] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 27 04:18:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:18:12.661765 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038e84c6-9ae8-4039-bb21-88397da647f6 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:12.662210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038e84c6-9ae8-4039-bb21-88397da647f6 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:12.662403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038e84c6-9ae8-4039-bb21-88397da647f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:12.681837 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-038e84c6-9ae8-4039-bb21-88397da647f6 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:18:12.683033 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-038e84c6-9ae8-4039-bb21-88397da647f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:18:12.691200 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 763/1523] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 27 04:18:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:18:12.847243 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10208173-0602-4c87-8cb7-6eff0252ef85 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:18:12.862650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10208173-0602-4c87-8cb7-6eff0252ef85 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:18:12.868014 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e42275e-53a8-44b3-8c1d-f9b99b6f617c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:18:12.868599 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e42275e-53a8-44b3-8c1d-f9b99b6f617c service nova] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:12.868871 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e42275e-53a8-44b3-8c1d-f9b99b6f617c service nova] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:12.869139 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e42275e-53a8-44b3-8c1d-f9b99b6f617c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:12.877566 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e42275e-53a8-44b3-8c1d-f9b99b6f617c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:18:12.879173 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e42275e-53a8-44b3-8c1d-f9b99b6f617c service nova] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:18:12.888593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 761/1524] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:18:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:18:13.949681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdeb284f-53be-4c22-939e-8dcbb8ffa068 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:13.950269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdeb284f-53be-4c22-939e-8dcbb8ffa068 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:13.950269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdeb284f-53be-4c22-939e-8dcbb8ffa068 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:13.952190 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 764/1525] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 27 04:18:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:18:13.958329 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07dc7e7-d268-4fe6-ae49-dfee9a2084e6 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:13.958739 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07dc7e7-d268-4fe6-ae49-dfee9a2084e6 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:13.958875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07dc7e7-d268-4fe6-ae49-dfee9a2084e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:13.977581 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-e07dc7e7-d268-4fe6-ae49-dfee9a2084e6 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:18:13.978790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-e07dc7e7-d268-4fe6-ae49-dfee9a2084e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:18:13.986604 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 762/1526] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 27 04:18:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:18:14.137794 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdeb284f-53be-4c22-939e-8dcbb8ffa068 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:18:14.152296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdeb284f-53be-4c22-939e-8dcbb8ffa068 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:18:14.157560 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2737bb72-e572-428f-9e2e-826e5729a911 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:18:14.158169 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2737bb72-e572-428f-9e2e-826e5729a911 service nova] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:14.158446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2737bb72-e572-428f-9e2e-826e5729a911 service nova] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:14.158711 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2737bb72-e572-428f-9e2e-826e5729a911 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:14.166598 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2737bb72-e572-428f-9e2e-826e5729a911 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:18:14.168157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2737bb72-e572-428f-9e2e-826e5729a911 service nova] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:18:14.177778 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 765/1527] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:18:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:18:18.104288 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd33bed-304d-4d80-9766-230e50f96d7f None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:18.104882 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd33bed-304d-4d80-9766-230e50f96d7f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:18.104882 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd33bed-304d-4d80-9766-230e50f96d7f None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:18.106217 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 763/1528] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 04:18:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:18:18.111815 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5e0f78-9ed3-4afb-90e6-a31784ca6aab None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:18.112238 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5e0f78-9ed3-4afb-90e6-a31784ca6aab None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:18.112363 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5e0f78-9ed3-4afb-90e6-a31784ca6aab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:18.132669 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-cc5e0f78-9ed3-4afb-90e6-a31784ca6aab None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:18:18.133883 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-cc5e0f78-9ed3-4afb-90e6-a31784ca6aab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:18:18.141523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 766/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 04:18:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:18:18.433274 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd33bed-304d-4d80-9766-230e50f96d7f None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:18:18.434985 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-dfd33bed-304d-4d80-9766-230e50f96d7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:18:18.510263 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd33bed-304d-4d80-9766-230e50f96d7f None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:18:18.513771 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c3b148-49f5-49b0-b929-8ad96b6670a3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:18:18.514376 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c3b148-49f5-49b0-b929-8ad96b6670a3 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:18.514654 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c3b148-49f5-49b0-b929-8ad96b6670a3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:18.514926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c3b148-49f5-49b0-b929-8ad96b6670a3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:18.523953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c3b148-49f5-49b0-b929-8ad96b6670a3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:18:18.525356 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c3b148-49f5-49b0-b929-8ad96b6670a3 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:18:18.535157 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 764/1530] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:18:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:18:20.661471 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2e8294-6619-4914-b2fe-1d6aafd3f0c4 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:20.662058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2e8294-6619-4914-b2fe-1d6aafd3f0c4 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:20.662058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2e8294-6619-4914-b2fe-1d6aafd3f0c4 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:20.663303 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 767/1531] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 04:18:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:18:20.668357 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ed9168-75b9-4fc8-ab56-8b73e639d751 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:20.668738 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ed9168-75b9-4fc8-ab56-8b73e639d751 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:20.668904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ed9168-75b9-4fc8-ab56-8b73e639d751 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:20.688270 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-51ed9168-75b9-4fc8-ab56-8b73e639d751 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:18:20.689429 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-51ed9168-75b9-4fc8-ab56-8b73e639d751 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:18:20.697080 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 765/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 04:18:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:18:20.984611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2e8294-6619-4914-b2fe-1d6aafd3f0c4 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:18:20.986322 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-bd2e8294-6619-4914-b2fe-1d6aafd3f0c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:18:20.993790 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=75 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:18:21.000618 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2e8294-6619-4914-b2fe-1d6aafd3f0c4 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:18:21.004182 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42d2040-199b-4057-ae39-7edace7d6a87 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:18:21.004776 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42d2040-199b-4057-ae39-7edace7d6a87 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:21.005042 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42d2040-199b-4057-ae39-7edace7d6a87 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:21.005313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42d2040-199b-4057-ae39-7edace7d6a87 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:21.014429 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42d2040-199b-4057-ae39-7edace7d6a87 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:18:21.015827 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42d2040-199b-4057-ae39-7edace7d6a87 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:18:21.025251 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 768/1533] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:18:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:18:24.554842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d27e180-fe13-4ccd-b7ba-87f6c8777d4a None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:24.555433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d27e180-fe13-4ccd-b7ba-87f6c8777d4a None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:24.555433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d27e180-fe13-4ccd-b7ba-87f6c8777d4a None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:24.556714 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 766/1534] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 04:18:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:18:24.562466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2a0747-95e1-4766-ae78-4b5c343bc1df None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:24.562880 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2a0747-95e1-4766-ae78-4b5c343bc1df None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:24.563077 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2a0747-95e1-4766-ae78-4b5c343bc1df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:24.581852 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-0e2a0747-95e1-4766-ae78-4b5c343bc1df None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:18:24.583041 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0e2a0747-95e1-4766-ae78-4b5c343bc1df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:18:24.591898 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 769/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 04:18:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:18:24.903841 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d27e180-fe13-4ccd-b7ba-87f6c8777d4a None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:18:24.906006 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-3d27e180-fe13-4ccd-b7ba-87f6c8777d4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:18:24.968331 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:18:24.981489 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d27e180-fe13-4ccd-b7ba-87f6c8777d4a None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:18:24.985542 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cbcc44-1945-4db1-bc78-37ba764294ed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:18:24.986166 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cbcc44-1945-4db1-bc78-37ba764294ed service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:24.986443 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cbcc44-1945-4db1-bc78-37ba764294ed service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:24.986714 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cbcc44-1945-4db1-bc78-37ba764294ed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:24.995345 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cbcc44-1945-4db1-bc78-37ba764294ed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:18:24.996757 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cbcc44-1945-4db1-bc78-37ba764294ed service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:18:25.005837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 767/1536] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:18:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:18:27.147245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432275d6-8a3a-4c6e-ab84-32c419ea4252 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:27.147774 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432275d6-8a3a-4c6e-ab84-32c419ea4252 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:27.147774 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432275d6-8a3a-4c6e-ab84-32c419ea4252 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:27.149273 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 770/1537] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 04:18:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:18:27.154979 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe6cde9-1361-4315-9c4f-4325985a57d7 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:27.155361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe6cde9-1361-4315-9c4f-4325985a57d7 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:27.155520 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe6cde9-1361-4315-9c4f-4325985a57d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:27.175065 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-1fe6cde9-1361-4315-9c4f-4325985a57d7 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:18:27.176244 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-1fe6cde9-1361-4315-9c4f-4325985a57d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:18:27.183881 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 768/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 04:18:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:18:27.484147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432275d6-8a3a-4c6e-ab84-32c419ea4252 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:18:27.485822 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-432275d6-8a3a-4c6e-ab84-32c419ea4252 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:18:27.498447 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432275d6-8a3a-4c6e-ab84-32c419ea4252 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:18:27.501891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97dd6158-7dde-447b-9bc1-d856edfdfee6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:18:27.502473 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dd6158-7dde-447b-9bc1-d856edfdfee6 service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:18:27.502747 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dd6158-7dde-447b-9bc1-d856edfdfee6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:18:27.503015 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dd6158-7dde-447b-9bc1-d856edfdfee6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:18:27.511575 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97dd6158-7dde-447b-9bc1-d856edfdfee6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:18:27.512962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97dd6158-7dde-447b-9bc1-d856edfdfee6 service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:18:27.521980 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 771/1539] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:18:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:18:37.169769 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=11 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:18:37.518011 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=14 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:19:52.765291 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6390847c-e2fc-4878-9887-70c669b2880d None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:52.766825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6390847c-e2fc-4878-9887-70c669b2880d None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:52.767355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6390847c-e2fc-4878-9887-70c669b2880d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:52.791433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-6390847c-e2fc-4878-9887-70c669b2880d None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:52.792656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6390847c-e2fc-4878-9887-70c669b2880d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:52.803956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 769/1540] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:52.812990 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28a9f6c-c65d-4fca-a59a-b28e8e301b49 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:52.813605 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28a9f6c-c65d-4fca-a59a-b28e8e301b49 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:52.813605 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28a9f6c-c65d-4fca-a59a-b28e8e301b49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:52.844368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-c28a9f6c-c65d-4fca-a59a-b28e8e301b49 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:52.845597 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c28a9f6c-c65d-4fca-a59a-b28e8e301b49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:52.857611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 772/1541] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:52.862746 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6390847c-e2fc-4878-9887-70c669b2880d None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:52.873095 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6390847c-e2fc-4878-9887-70c669b2880d None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:52.881114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508c4c4e-1ca7-4684-8f72-bb91614a8f1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:52.881893 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508c4c4e-1ca7-4684-8f72-bb91614a8f1b admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:52.882177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508c4c4e-1ca7-4684-8f72-bb91614a8f1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:52.882451 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508c4c4e-1ca7-4684-8f72-bb91614a8f1b admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:52.883103 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508c4c4e-1ca7-4684-8f72-bb91614a8f1b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:19:52.883402 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508c4c4e-1ca7-4684-8f72-bb91614a8f1b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:52.885004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508c4c4e-1ca7-4684-8f72-bb91614a8f1b admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:52.902049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 770/1542] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:19:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:19:52.910967 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28a9f6c-c65d-4fca-a59a-b28e8e301b49 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:52.930239 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28a9f6c-c65d-4fca-a59a-b28e8e301b49 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:52.935159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819552f7-dec6-441f-aeab-f9ac43153eb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:52.935783 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819552f7-dec6-441f-aeab-f9ac43153eb8 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:52.936233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819552f7-dec6-441f-aeab-f9ac43153eb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:52.936511 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819552f7-dec6-441f-aeab-f9ac43153eb8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:52.937156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819552f7-dec6-441f-aeab-f9ac43153eb8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:52.939069 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819552f7-dec6-441f-aeab-f9ac43153eb8 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:52.996671 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47db1e23-3928-4db6-81f6-4aa816fe84a2 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:52.997047 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47db1e23-3928-4db6-81f6-4aa816fe84a2 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:52.997219 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47db1e23-3928-4db6-81f6-4aa816fe84a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:52.997617 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 773/1543] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:19:52 2022] POST /identity/v3/projects => generated 377 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:19:53.001623 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819552f7-dec6-441f-aeab-f9ac43153eb8 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:53.037323 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-47db1e23-3928-4db6-81f6-4aa816fe84a2 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:53.038491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-47db1e23-3928-4db6-81f6-4aa816fe84a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:53.112533 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819552f7-dec6-441f-aeab-f9ac43153eb8 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:53.121776 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8ffb4d-e045-4a5b-a067-7327943b7295 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:53.122434 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8ffb4d-e045-4a5b-a067-7327943b7295 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.122904 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8ffb4d-e045-4a5b-a067-7327943b7295 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.123189 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8ffb4d-e045-4a5b-a067-7327943b7295 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.124823 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8ffb4d-e045-4a5b-a067-7327943b7295 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:53.125937 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: WARNING py.warnings [None req-ed8ffb4d-e045-4a5b-a067-7327943b7295 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 27 04:19:53.125937 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: warnings.warn(msg) Sep 27 04:19:53.125937 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: Sep 27 04:19:53.127171 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8ffb4d-e045-4a5b-a067-7327943b7295 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:53.172093 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 774/1544] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:19:53 2022] POST /identity/v3/users => generated 289 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:19:53.175074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df5926b-8988-499b-abb1-d9655a353b0b None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.175362 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df5926b-8988-499b-abb1-d9655a353b0b None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.176850 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 771/1545] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:19:53.184154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df5926b-8988-499b-abb1-d9655a353b0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.189203 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03cab37-ac90-407c-9c3d-eb56b7aa08a8 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.189503 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03cab37-ac90-407c-9c3d-eb56b7aa08a8 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.189779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03cab37-ac90-407c-9c3d-eb56b7aa08a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.234669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-d03cab37-ac90-407c-9c3d-eb56b7aa08a8 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:53.235496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-5df5926b-8988-499b-abb1-d9655a353b0b None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:53.237473 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-5df5926b-8988-499b-abb1-d9655a353b0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:53.238039 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d03cab37-ac90-407c-9c3d-eb56b7aa08a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:53.255187 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 772/1546] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:53.256782 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03cab37-ac90-407c-9c3d-eb56b7aa08a8 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:53.257129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 775/1547] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:53.258420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31830dc4-75a7-4e0b-8043-4d38dba88713 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.258705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31830dc4-75a7-4e0b-8043-4d38dba88713 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.258975 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31830dc4-75a7-4e0b-8043-4d38dba88713 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.280561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03cab37-ac90-407c-9c3d-eb56b7aa08a8 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:53.290317 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2147e5ad-db6d-43c9-9be4-57f8c8968eaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:53.290854 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2147e5ad-db6d-43c9-9be4-57f8c8968eaa admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.291129 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2147e5ad-db6d-43c9-9be4-57f8c8968eaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.291401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2147e5ad-db6d-43c9-9be4-57f8c8968eaa admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.292049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2147e5ad-db6d-43c9-9be4-57f8c8968eaa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:53.293643 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2147e5ad-db6d-43c9-9be4-57f8c8968eaa admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:53.294295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-31830dc4-75a7-4e0b-8043-4d38dba88713 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:53.295637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-31830dc4-75a7-4e0b-8043-4d38dba88713 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:53.309006 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 776/1548] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:53.310305 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ed0620-8f86-4efa-b89b-2ec2ff98a758 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.310564 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ed0620-8f86-4efa-b89b-2ec2ff98a758 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.310837 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ed0620-8f86-4efa-b89b-2ec2ff98a758 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.316924 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 773/1549] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:19:53 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:19:53.319879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22f73fd-dd27-45d8-8508-0cea74b7f31c None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.320015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22f73fd-dd27-45d8-8508-0cea74b7f31c None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.320259 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22f73fd-dd27-45d8-8508-0cea74b7f31c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.344044 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-16ed0620-8f86-4efa-b89b-2ec2ff98a758 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:53.345281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-c22f73fd-dd27-45d8-8508-0cea74b7f31c None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:53.346065 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-16ed0620-8f86-4efa-b89b-2ec2ff98a758 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:53.346550 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c22f73fd-dd27-45d8-8508-0cea74b7f31c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:53.358663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 774/1550] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:53.358663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3d0fe5-9a7e-4d41-8ce2-e5578d4518d0 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.358663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3d0fe5-9a7e-4d41-8ce2-e5578d4518d0 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.358663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3d0fe5-9a7e-4d41-8ce2-e5578d4518d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.360355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 777/1551] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:53.361842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ed0620-8f86-4efa-b89b-2ec2ff98a758 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:53.375647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ed0620-8f86-4efa-b89b-2ec2ff98a758 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:53.380823 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5ca3fa-e5c9-4222-9c12-c03e72683ade admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:53.381803 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5ca3fa-e5c9-4222-9c12-c03e72683ade admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.382401 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5ca3fa-e5c9-4222-9c12-c03e72683ade admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.382592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-7c3d0fe5-9a7e-4d41-8ce2-e5578d4518d0 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:53.383067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5ca3fa-e5c9-4222-9c12-c03e72683ade admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.383783 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-7c3d0fe5-9a7e-4d41-8ce2-e5578d4518d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:53.384263 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5ca3fa-e5c9-4222-9c12-c03e72683ade admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:19:53.384879 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5ca3fa-e5c9-4222-9c12-c03e72683ade admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:53.386201 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: WARNING py.warnings [None req-da5ca3fa-e5c9-4222-9c12-c03e72683ade admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 27 04:19:53.386201 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: warnings.warn(msg) Sep 27 04:19:53.386201 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: Sep 27 04:19:53.387684 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5ca3fa-e5c9-4222-9c12-c03e72683ade admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:53.392654 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 775/1552] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:53.393449 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3d0fe5-9a7e-4d41-8ce2-e5578d4518d0 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:53.402739 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 778/1553] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:19:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:19:53.404714 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d6d863-79f9-40ca-844e-f42226b0bdbb None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.405280 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d6d863-79f9-40ca-844e-f42226b0bdbb None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.405782 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3d0fe5-9a7e-4d41-8ce2-e5578d4518d0 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:53.406079 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d6d863-79f9-40ca-844e-f42226b0bdbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.415652 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c816acd4-d80c-42b9-b2f8-ccb2feb5607c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:53.416559 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c816acd4-d80c-42b9-b2f8-ccb2feb5607c admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.416849 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c816acd4-d80c-42b9-b2f8-ccb2feb5607c admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.417129 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c816acd4-d80c-42b9-b2f8-ccb2feb5607c admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.417811 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c816acd4-d80c-42b9-b2f8-ccb2feb5607c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:53.420209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c816acd4-d80c-42b9-b2f8-ccb2feb5607c admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:53.432850 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-76d6d863-79f9-40ca-844e-f42226b0bdbb None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:53.433215 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 776/1554] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:19:53 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:19:53.434074 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c816acd4-d80c-42b9-b2f8-ccb2feb5607c admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:53.434893 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-76d6d863-79f9-40ca-844e-f42226b0bdbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:53.447821 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 779/1555] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:53.449149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d6d863-79f9-40ca-844e-f42226b0bdbb None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:53.453851 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c816acd4-d80c-42b9-b2f8-ccb2feb5607c admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:53.458734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c2810e-e772-4eaa-967b-d9448e58cc77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:53.459332 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c2810e-e772-4eaa-967b-d9448e58cc77 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.459458 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d6d863-79f9-40ca-844e-f42226b0bdbb None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:53.459609 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c2810e-e772-4eaa-967b-d9448e58cc77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.459916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c2810e-e772-4eaa-967b-d9448e58cc77 admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.460588 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c2810e-e772-4eaa-967b-d9448e58cc77 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:19:53.460884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c2810e-e772-4eaa-967b-d9448e58cc77 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:53.462588 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c2810e-e772-4eaa-967b-d9448e58cc77 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:53.463881 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a82453-1570-43d6-b8e9-7f17c21399d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:53.464643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a82453-1570-43d6-b8e9-7f17c21399d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.465386 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a82453-1570-43d6-b8e9-7f17c21399d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.465668 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a82453-1570-43d6-b8e9-7f17c21399d9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.466336 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a82453-1570-43d6-b8e9-7f17c21399d9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:19:53.466638 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a82453-1570-43d6-b8e9-7f17c21399d9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:53.468223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a82453-1570-43d6-b8e9-7f17c21399d9 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:53.474697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 777/1556] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:19:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:19:53.475898 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7df952-2f33-4c73-8910-b636f595958b None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.476179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7df952-2f33-4c73-8910-b636f595958b None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.476452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7df952-2f33-4c73-8910-b636f595958b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.484036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 780/1557] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:19:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:19:53.484036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93cd5dc-612d-44d4-9946-e019015a8b90 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.484036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93cd5dc-612d-44d4-9946-e019015a8b90 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.484036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93cd5dc-612d-44d4-9946-e019015a8b90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.498147 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-3c7df952-2f33-4c73-8910-b636f595958b None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:53.500048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-3c7df952-2f33-4c73-8910-b636f595958b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:53.502442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-c93cd5dc-612d-44d4-9946-e019015a8b90 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:53.503591 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c93cd5dc-612d-44d4-9946-e019015a8b90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:53.511952 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 781/1558] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:53.512674 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93cd5dc-612d-44d4-9946-e019015a8b90 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:53.513349 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 778/1559] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:53.514087 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7df952-2f33-4c73-8910-b636f595958b None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:53.528366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93cd5dc-612d-44d4-9946-e019015a8b90 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:53.528573 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7df952-2f33-4c73-8910-b636f595958b None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:53.529146 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5e208c-bfca-44e2-8e5f-24bb08b7b4f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:53.529755 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5e208c-bfca-44e2-8e5f-24bb08b7b4f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.530046 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5e208c-bfca-44e2-8e5f-24bb08b7b4f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.530352 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5e208c-bfca-44e2-8e5f-24bb08b7b4f1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.530988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5e208c-bfca-44e2-8e5f-24bb08b7b4f1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:53.531778 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8090ed9-3cfd-4f6b-8da2-ee0fa6e1ed7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:53.532446 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8090ed9-3cfd-4f6b-8da2-ee0fa6e1ed7b admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.532720 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8090ed9-3cfd-4f6b-8da2-ee0fa6e1ed7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.532913 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5e208c-bfca-44e2-8e5f-24bb08b7b4f1 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:53.533094 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8090ed9-3cfd-4f6b-8da2-ee0fa6e1ed7b admin admin] PATH_INFO: `/v3/projects/1c6cc70cbd2b4c04864f33ebafa06b5a/users/f3438d850ce541cb9027ea3d5fe7dbf3/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.550717 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8090ed9-3cfd-4f6b-8da2-ee0fa6e1ed7b admin admin] RBAC: Authorizing `identity:create_grant(project_id=1c6cc70cbd2b4c04864f33ebafa06b5a, user_id=f3438d850ce541cb9027ea3d5fe7dbf3, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:53.551835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: WARNING py.warnings [None req-f8090ed9-3cfd-4f6b-8da2-ee0fa6e1ed7b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 27 04:19:53.551835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: warnings.warn(msg) Sep 27 04:19:53.551835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: Sep 27 04:19:53.554070 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8090ed9-3cfd-4f6b-8da2-ee0fa6e1ed7b admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:53.566609 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 782/1560] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:19:53 2022] POST /identity/v3/projects => generated 389 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:19:53.566609 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1ac7e8-c4db-48e3-8dc2-e13414474827 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.566609 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1ac7e8-c4db-48e3-8dc2-e13414474827 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.566609 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1ac7e8-c4db-48e3-8dc2-e13414474827 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.570285 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 779/1561] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:19:53 2022] PUT /identity/v3/projects/1c6cc70cbd2b4c04864f33ebafa06b5a/users/f3438d850ce541cb9027ea3d5fe7dbf3/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:19:53.571370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3b6419-e724-4c90-8640-05ea4015ce3f None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.571652 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3b6419-e724-4c90-8640-05ea4015ce3f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.571926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3b6419-e724-4c90-8640-05ea4015ce3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.587856 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-cc1ac7e8-c4db-48e3-8dc2-e13414474827 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:53.589046 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-cc1ac7e8-c4db-48e3-8dc2-e13414474827 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:53.590125 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-5d3b6419-e724-4c90-8640-05ea4015ce3f None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:53.591277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-5d3b6419-e724-4c90-8640-05ea4015ce3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:53.616044 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:19:53.639803 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:19:53.650471 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 783/1562] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:53.651552 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1ac7e8-c4db-48e3-8dc2-e13414474827 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:53.653292 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 780/1563] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:53.658148 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3b6419-e724-4c90-8640-05ea4015ce3f None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:53.663958 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1ac7e8-c4db-48e3-8dc2-e13414474827 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:53.668856 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77580629-c256-45c3-b463-a0cf3ee9ddef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:53.669464 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77580629-c256-45c3-b463-a0cf3ee9ddef admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.669752 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77580629-c256-45c3-b463-a0cf3ee9ddef admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.670233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77580629-c256-45c3-b463-a0cf3ee9ddef admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.671297 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77580629-c256-45c3-b463-a0cf3ee9ddef admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:53.673276 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77580629-c256-45c3-b463-a0cf3ee9ddef admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:53.673664 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3b6419-e724-4c90-8640-05ea4015ce3f None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:53.679567 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb53909-f09c-4256-a9dd-081ee93df8fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:53.679938 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb53909-f09c-4256-a9dd-081ee93df8fe admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.680211 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb53909-f09c-4256-a9dd-081ee93df8fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.680477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb53909-f09c-4256-a9dd-081ee93df8fe admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.681231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb53909-f09c-4256-a9dd-081ee93df8fe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:53.682334 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: WARNING py.warnings [None req-edb53909-f09c-4256-a9dd-081ee93df8fe admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 27 04:19:53.682334 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: warnings.warn(msg) Sep 27 04:19:53.682334 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: Sep 27 04:19:53.683709 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb53909-f09c-4256-a9dd-081ee93df8fe admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:53.702372 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 784/1564] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:19:53 2022] POST /identity/v3/projects => generated 379 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:19:53.703277 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77580629-c256-45c3-b463-a0cf3ee9ddef admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:53.712107 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 781/1565] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:19:53 2022] POST /identity/v3/projects => generated 367 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:19:53.715713 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22043a5-0fcf-4c7e-b50b-53df4b882f02 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.715713 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22043a5-0fcf-4c7e-b50b-53df4b882f02 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.715713 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22043a5-0fcf-4c7e-b50b-53df4b882f02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.732217 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-f22043a5-0fcf-4c7e-b50b-53df4b882f02 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:53.733919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-f22043a5-0fcf-4c7e-b50b-53df4b882f02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:53.764465 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77580629-c256-45c3-b463-a0cf3ee9ddef admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:53.793618 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 782/1566] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:53.794598 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22043a5-0fcf-4c7e-b50b-53df4b882f02 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:53.806882 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22043a5-0fcf-4c7e-b50b-53df4b882f02 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:53.811345 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaec8755-9858-4567-a20b-682656b5c314 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:53.811959 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaec8755-9858-4567-a20b-682656b5c314 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.812235 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaec8755-9858-4567-a20b-682656b5c314 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.812506 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaec8755-9858-4567-a20b-682656b5c314 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.813123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaec8755-9858-4567-a20b-682656b5c314 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:53.815054 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaec8755-9858-4567-a20b-682656b5c314 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:53.820320 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631b119a-e5ea-420c-90b3-221794a15a5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:53.820924 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631b119a-e5ea-420c-90b3-221794a15a5f admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.821202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631b119a-e5ea-420c-90b3-221794a15a5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.821472 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631b119a-e5ea-420c-90b3-221794a15a5f admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.822152 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-631b119a-e5ea-420c-90b3-221794a15a5f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:19:53.822457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-631b119a-e5ea-420c-90b3-221794a15a5f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:53.825089 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-631b119a-e5ea-420c-90b3-221794a15a5f admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:53.835879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 783/1567] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:19:53 2022] POST /identity/v3/users => generated 295 bytes in 42 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:19:53.838912 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 785/1568] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:19:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:19:53.839767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631b119a-e5ea-420c-90b3-221794a15a5f admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:53.840338 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd18475d-fbf4-433b-8bbb-3d6dff1338bc None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.840646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd18475d-fbf4-433b-8bbb-3d6dff1338bc None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.840892 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd18475d-fbf4-433b-8bbb-3d6dff1338bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.850363 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631b119a-e5ea-420c-90b3-221794a15a5f admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:53.855699 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5a571b-ca98-425d-afc9-3dbcfafd52bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:53.856315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5a571b-ca98-425d-afc9-3dbcfafd52bf admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.856598 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5a571b-ca98-425d-afc9-3dbcfafd52bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.856872 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5a571b-ca98-425d-afc9-3dbcfafd52bf admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.857540 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5a571b-ca98-425d-afc9-3dbcfafd52bf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:19:53.857882 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5a571b-ca98-425d-afc9-3dbcfafd52bf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:53.860076 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5a571b-ca98-425d-afc9-3dbcfafd52bf admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:53.863691 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-bd18475d-fbf4-433b-8bbb-3d6dff1338bc None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:53.864873 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-bd18475d-fbf4-433b-8bbb-3d6dff1338bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:53.872555 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 786/1569] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:19:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:19:53.872790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 784/1570] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:53.873769 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f0a0b7-b4e5-4207-99b9-9e97ca61e89b None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.874080 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f0a0b7-b4e5-4207-99b9-9e97ca61e89b None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.874205 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd18475d-fbf4-433b-8bbb-3d6dff1338bc None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:53.874352 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f0a0b7-b4e5-4207-99b9-9e97ca61e89b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.888227 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd18475d-fbf4-433b-8bbb-3d6dff1338bc None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:53.892804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f70b8a8-a88c-43f5-a2be-67f4e1acd51c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:53.893394 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f70b8a8-a88c-43f5-a2be-67f4e1acd51c admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.893664 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f70b8a8-a88c-43f5-a2be-67f4e1acd51c admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.894031 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f70b8a8-a88c-43f5-a2be-67f4e1acd51c admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.894650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f70b8a8-a88c-43f5-a2be-67f4e1acd51c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:53.895748 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-94f0a0b7-b4e5-4207-99b9-9e97ca61e89b None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:53.896576 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f70b8a8-a88c-43f5-a2be-67f4e1acd51c admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:53.896962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-94f0a0b7-b4e5-4207-99b9-9e97ca61e89b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:53.905711 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 787/1571] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:53.906348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f0a0b7-b4e5-4207-99b9-9e97ca61e89b None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:53.915677 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f0a0b7-b4e5-4207-99b9-9e97ca61e89b None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:53.920356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a172ab14-7071-4bc2-9337-11707af5ef9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:53.921572 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a172ab14-7071-4bc2-9337-11707af5ef9f admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.921882 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a172ab14-7071-4bc2-9337-11707af5ef9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.922158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a172ab14-7071-4bc2-9337-11707af5ef9f admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.922781 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a172ab14-7071-4bc2-9337-11707af5ef9f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:53.927166 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 785/1572] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:19:53 2022] POST /identity/v3/users => generated 290 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:19:53.927320 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a172ab14-7071-4bc2-9337-11707af5ef9f admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:53.927433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f70b8a8-a88c-43f5-a2be-67f4e1acd51c admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:53.935020 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f70b8a8-a88c-43f5-a2be-67f4e1acd51c admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:53.940745 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087b55b6-78e9-4902-bd2e-dd4188d4261a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:53.941412 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087b55b6-78e9-4902-bd2e-dd4188d4261a admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.941766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087b55b6-78e9-4902-bd2e-dd4188d4261a admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.942072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087b55b6-78e9-4902-bd2e-dd4188d4261a admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.942695 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087b55b6-78e9-4902-bd2e-dd4188d4261a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:19:53.942993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087b55b6-78e9-4902-bd2e-dd4188d4261a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:53.944564 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087b55b6-78e9-4902-bd2e-dd4188d4261a admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:53.949068 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 788/1573] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:19:53 2022] POST /identity/v3/users => generated 284 bytes in 40 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:19:53.949068 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a172ab14-7071-4bc2-9337-11707af5ef9f admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:53.957522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a172ab14-7071-4bc2-9337-11707af5ef9f admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:53.957682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 786/1574] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:19:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:19:53.958461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087b55b6-78e9-4902-bd2e-dd4188d4261a admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:53.967639 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087b55b6-78e9-4902-bd2e-dd4188d4261a admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:53.969649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8ab965-02b0-4e13-b936-39d91ceed245 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:53.970355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8ab965-02b0-4e13-b936-39d91ceed245 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.970697 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8ab965-02b0-4e13-b936-39d91ceed245 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.971042 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8ab965-02b0-4e13-b936-39d91ceed245 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.971736 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8ab965-02b0-4e13-b936-39d91ceed245 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:53.972814 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90d26d3-f549-4213-b220-012c148fbe0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:53.973384 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90d26d3-f549-4213-b220-012c148fbe0c admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:53.973657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90d26d3-f549-4213-b220-012c148fbe0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:53.973785 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8ab965-02b0-4e13-b936-39d91ceed245 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:53.974034 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90d26d3-f549-4213-b220-012c148fbe0c admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:53.974664 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e90d26d3-f549-4213-b220-012c148fbe0c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:53.976197 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e90d26d3-f549-4213-b220-012c148fbe0c admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:53.985278 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:19:53.986334 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 787/1575] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:19:53 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:19:53.986937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90d26d3-f549-4213-b220-012c148fbe0c admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:53.997102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90d26d3-f549-4213-b220-012c148fbe0c admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:54.002314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2855eb3c-7ed4-4baf-bed9-8322093b285e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:54.002923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2855eb3c-7ed4-4baf-bed9-8322093b285e admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.003644 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2855eb3c-7ed4-4baf-bed9-8322093b285e admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.003929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2855eb3c-7ed4-4baf-bed9-8322093b285e admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.004544 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2855eb3c-7ed4-4baf-bed9-8322093b285e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:54.006455 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2855eb3c-7ed4-4baf-bed9-8322093b285e admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:54.009899 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 789/1576] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:19:53 2022] POST /identity/v3/projects => generated 363 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:19:54.011172 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8ab965-02b0-4e13-b936-39d91ceed245 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:54.039336 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 788/1577] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:19:53 2022] POST /identity/v3/projects => generated 369 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:19:54.039336 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2855eb3c-7ed4-4baf-bed9-8322093b285e admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:54.070071 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8ab965-02b0-4e13-b936-39d91ceed245 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:54.086977 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:19:54.101686 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2855eb3c-7ed4-4baf-bed9-8322093b285e admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:54.106391 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfcf2b6-8cfd-4082-b63f-762429c77a88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:54.107032 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfcf2b6-8cfd-4082-b63f-762429c77a88 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.107313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfcf2b6-8cfd-4082-b63f-762429c77a88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.107575 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfcf2b6-8cfd-4082-b63f-762429c77a88 admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.108229 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcfcf2b6-8cfd-4082-b63f-762429c77a88 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:19:54.108566 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcfcf2b6-8cfd-4082-b63f-762429c77a88 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:54.110203 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcfcf2b6-8cfd-4082-b63f-762429c77a88 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:54.118387 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de59231c-dfd0-44c3-9ec7-0e571bc72141 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:54.119538 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de59231c-dfd0-44c3-9ec7-0e571bc72141 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.119980 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de59231c-dfd0-44c3-9ec7-0e571bc72141 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.120364 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de59231c-dfd0-44c3-9ec7-0e571bc72141 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.121776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 789/1578] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:19:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:19:54.122594 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfcf2b6-8cfd-4082-b63f-762429c77a88 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:54.132633 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de59231c-dfd0-44c3-9ec7-0e571bc72141 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:54.133955 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfcf2b6-8cfd-4082-b63f-762429c77a88 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:54.134173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de59231c-dfd0-44c3-9ec7-0e571bc72141 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:54.138415 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf29c98f-4725-4ae6-b382-0c087c790349 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:54.138993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf29c98f-4725-4ae6-b382-0c087c790349 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.139262 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf29c98f-4725-4ae6-b382-0c087c790349 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.139554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf29c98f-4725-4ae6-b382-0c087c790349 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.140815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf29c98f-4725-4ae6-b382-0c087c790349 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:54.142412 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf29c98f-4725-4ae6-b382-0c087c790349 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:54.144796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 790/1579] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:19:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:19:54.145094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de59231c-dfd0-44c3-9ec7-0e571bc72141 service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:54.152949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 790/1580] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:19:54 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:19:54.153794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf29c98f-4725-4ae6-b382-0c087c790349 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:54.154319 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de59231c-dfd0-44c3-9ec7-0e571bc72141 service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:54.159299 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5eb88a-5070-4e2e-ab73-6b066cbf8dea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:54.159894 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5eb88a-5070-4e2e-ab73-6b066cbf8dea admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.160178 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5eb88a-5070-4e2e-ab73-6b066cbf8dea admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.160454 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5eb88a-5070-4e2e-ab73-6b066cbf8dea admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.161225 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5eb88a-5070-4e2e-ab73-6b066cbf8dea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:54.162823 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5eb88a-5070-4e2e-ab73-6b066cbf8dea admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:54.163753 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf29c98f-4725-4ae6-b382-0c087c790349 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:54.169055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2606d408-178d-45dc-8d30-99d40fd08709 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:54.169641 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2606d408-178d-45dc-8d30-99d40fd08709 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.169957 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2606d408-178d-45dc-8d30-99d40fd08709 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.170227 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2606d408-178d-45dc-8d30-99d40fd08709 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.171398 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2606d408-178d-45dc-8d30-99d40fd08709 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:54.173306 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2606d408-178d-45dc-8d30-99d40fd08709 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:54.175991 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 791/1581] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:19:54 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:19:54.176954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5eb88a-5070-4e2e-ab73-6b066cbf8dea admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:54.189237 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5eb88a-5070-4e2e-ab73-6b066cbf8dea admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:54.195298 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30761552-947b-42a9-9c2c-b0e59d45df8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:54.195898 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30761552-947b-42a9-9c2c-b0e59d45df8a admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.196239 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30761552-947b-42a9-9c2c-b0e59d45df8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.196441 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30761552-947b-42a9-9c2c-b0e59d45df8a admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.197082 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30761552-947b-42a9-9c2c-b0e59d45df8a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:54.198690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30761552-947b-42a9-9c2c-b0e59d45df8a admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:54.205991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 791/1582] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:19:54 2022] POST /identity/v3/projects => generated 355 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:19:54.207270 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2606d408-178d-45dc-8d30-99d40fd08709 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:54.210413 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 792/1583] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:19:54 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:19:54.211548 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30761552-947b-42a9-9c2c-b0e59d45df8a admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:54.224148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30761552-947b-42a9-9c2c-b0e59d45df8a admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:54.282523 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2606d408-178d-45dc-8d30-99d40fd08709 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:54.287680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e321b5-0b12-40cd-b919-533e98efb2d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:54.288280 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e321b5-0b12-40cd-b919-533e98efb2d3 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.288569 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ae5d71-01d0-48f9-b313-92ba6e35556e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:54.288706 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e321b5-0b12-40cd-b919-533e98efb2d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.288800 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e321b5-0b12-40cd-b919-533e98efb2d3 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.289380 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ae5d71-01d0-48f9-b313-92ba6e35556e admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.289495 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e321b5-0b12-40cd-b919-533e98efb2d3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:54.289586 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ae5d71-01d0-48f9-b313-92ba6e35556e admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.289762 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ae5d71-01d0-48f9-b313-92ba6e35556e admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.290418 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ae5d71-01d0-48f9-b313-92ba6e35556e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:54.291348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e321b5-0b12-40cd-b919-533e98efb2d3 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:54.292583 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ae5d71-01d0-48f9-b313-92ba6e35556e admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:54.315029 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 792/1584] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:19:54 2022] POST /identity/v3/users => generated 282 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:19:54.315835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e321b5-0b12-40cd-b919-533e98efb2d3 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:54.321849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 793/1585] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:19:54 2022] POST /identity/v3/users => generated 285 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:19:54.322809 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ae5d71-01d0-48f9-b313-92ba6e35556e admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:54.330156 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e321b5-0b12-40cd-b919-533e98efb2d3 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:54.334606 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ae5d71-01d0-48f9-b313-92ba6e35556e admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:54.334860 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2d3668-4855-4fa6-9a31-604fbecacd34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:54.335466 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2d3668-4855-4fa6-9a31-604fbecacd34 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.335739 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2d3668-4855-4fa6-9a31-604fbecacd34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.336051 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2d3668-4855-4fa6-9a31-604fbecacd34 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.336684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2d3668-4855-4fa6-9a31-604fbecacd34 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:54.338628 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2d3668-4855-4fa6-9a31-604fbecacd34 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:54.340427 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea723e9c-8c67-4829-83eb-97dd59233c94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:54.341099 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea723e9c-8c67-4829-83eb-97dd59233c94 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.341569 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea723e9c-8c67-4829-83eb-97dd59233c94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.341691 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea723e9c-8c67-4829-83eb-97dd59233c94 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.342562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea723e9c-8c67-4829-83eb-97dd59233c94 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:54.344187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea723e9c-8c67-4829-83eb-97dd59233c94 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:54.355705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 794/1586] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:19:54 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:19:54.356517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea723e9c-8c67-4829-83eb-97dd59233c94 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:54.368094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea723e9c-8c67-4829-83eb-97dd59233c94 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:54.374887 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1043007-1108-4c7d-b45d-8d65a35d62c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:54.375250 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1043007-1108-4c7d-b45d-8d65a35d62c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.375914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1043007-1108-4c7d-b45d-8d65a35d62c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.376204 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1043007-1108-4c7d-b45d-8d65a35d62c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.377051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1043007-1108-4c7d-b45d-8d65a35d62c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:54.378802 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1043007-1108-4c7d-b45d-8d65a35d62c1 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:54.385849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:19:54.386054 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 793/1587] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:19:54 2022] POST /identity/v3/projects => generated 381 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:19:54.388523 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2d3668-4855-4fa6-9a31-604fbecacd34 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:54.392658 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 795/1588] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:19:54 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:19:54.394626 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1043007-1108-4c7d-b45d-8d65a35d62c1 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:54.406988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1043007-1108-4c7d-b45d-8d65a35d62c1 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:54.458955 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2d3668-4855-4fa6-9a31-604fbecacd34 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:54.463697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88978475-c64d-43ed-a963-c390226c9dd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:54.464397 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88978475-c64d-43ed-a963-c390226c9dd8 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.464735 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88978475-c64d-43ed-a963-c390226c9dd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.465100 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88978475-c64d-43ed-a963-c390226c9dd8 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.465816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88978475-c64d-43ed-a963-c390226c9dd8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:54.467812 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88978475-c64d-43ed-a963-c390226c9dd8 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:54.473474 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b0052b-5416-4dd8-be9b-740d73c025f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:54.474312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b0052b-5416-4dd8-be9b-740d73c025f0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.474582 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b0052b-5416-4dd8-be9b-740d73c025f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.475025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b0052b-5416-4dd8-be9b-740d73c025f0 admin admin] PATH_INFO: `/v3/projects/312fa89a4d9b43c5919b7bad5411002a/users/7c56e7333ce4450b8cbd5caeafc73947/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.494449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b0052b-5416-4dd8-be9b-740d73c025f0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=312fa89a4d9b43c5919b7bad5411002a, user_id=7c56e7333ce4450b8cbd5caeafc73947, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:54.494997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 794/1589] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:19:54 2022] POST /identity/v3/users => generated 278 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:19:54.497541 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b0052b-5416-4dd8-be9b-740d73c025f0 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:54.501952 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88978475-c64d-43ed-a963-c390226c9dd8 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:54.509517 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:19:54.514063 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88978475-c64d-43ed-a963-c390226c9dd8 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:54.516672 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 796/1590] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:19:54 2022] PUT /identity/v3/projects/312fa89a4d9b43c5919b7bad5411002a/users/7c56e7333ce4450b8cbd5caeafc73947/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:19:54.517389 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b0052b-5416-4dd8-be9b-740d73c025f0 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:54.597862 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70a9aa7-ae0a-4ebe-94cb-bd2a89b04bca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:54.599710 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70a9aa7-ae0a-4ebe-94cb-bd2a89b04bca admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.600079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70a9aa7-ae0a-4ebe-94cb-bd2a89b04bca admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.600440 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70a9aa7-ae0a-4ebe-94cb-bd2a89b04bca admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.601329 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70a9aa7-ae0a-4ebe-94cb-bd2a89b04bca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:54.602637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b0052b-5416-4dd8-be9b-740d73c025f0 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:54.603176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70a9aa7-ae0a-4ebe-94cb-bd2a89b04bca admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:54.607819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d364f61-5249-422d-9afd-f51e204cc35a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:54.609048 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d364f61-5249-422d-9afd-f51e204cc35a admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.609561 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d364f61-5249-422d-9afd-f51e204cc35a admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.610142 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d364f61-5249-422d-9afd-f51e204cc35a admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.611277 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d364f61-5249-422d-9afd-f51e204cc35a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:54.613443 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d364f61-5249-422d-9afd-f51e204cc35a admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:54.615463 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 795/1591] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:19:54 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:19:54.616269 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70a9aa7-ae0a-4ebe-94cb-bd2a89b04bca admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:54.628197 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70a9aa7-ae0a-4ebe-94cb-bd2a89b04bca admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:54.628834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 797/1592] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:19:54 2022] GET /identity/v3/roles => generated 1420 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:19:54.629912 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d364f61-5249-422d-9afd-f51e204cc35a admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:54.633169 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb2ffab-b039-4436-8404-440f02791cd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:54.633972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb2ffab-b039-4436-8404-440f02791cd6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.634345 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb2ffab-b039-4436-8404-440f02791cd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.634729 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb2ffab-b039-4436-8404-440f02791cd6 admin admin] PATH_INFO: `/v3/projects/7914c4b5217f411f807854ccb07e0365/users/03912e6e8a074eb1814c90158c246281/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.639900 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d364f61-5249-422d-9afd-f51e204cc35a admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:54.644551 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b7ecd2-4d7b-4595-a371-66362b40816f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:54.645146 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b7ecd2-4d7b-4595-a371-66362b40816f admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.645427 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b7ecd2-4d7b-4595-a371-66362b40816f admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.645700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b7ecd2-4d7b-4595-a371-66362b40816f admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.646359 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b7ecd2-4d7b-4595-a371-66362b40816f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:54.648265 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b7ecd2-4d7b-4595-a371-66362b40816f admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:54.650495 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb2ffab-b039-4436-8404-440f02791cd6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7914c4b5217f411f807854ccb07e0365, user_id=03912e6e8a074eb1814c90158c246281, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:54.653463 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb2ffab-b039-4436-8404-440f02791cd6 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:54.671822 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 796/1593] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:19:54 2022] PUT /identity/v3/projects/7914c4b5217f411f807854ccb07e0365/users/03912e6e8a074eb1814c90158c246281/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:19:54.673121 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb2ffab-b039-4436-8404-440f02791cd6 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:54.676761 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 798/1594] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:19:54 2022] POST /identity/v3/users => generated 291 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:19:54.676761 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b7ecd2-4d7b-4595-a371-66362b40816f admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:54.687534 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b7ecd2-4d7b-4595-a371-66362b40816f admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:54.743300 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb2ffab-b039-4436-8404-440f02791cd6 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:54.743697 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:19:54.748916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca02a24d-843d-4967-a6b8-4be90d58b124 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:54.749584 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca02a24d-843d-4967-a6b8-4be90d58b124 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.749900 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca02a24d-843d-4967-a6b8-4be90d58b124 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.750285 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca02a24d-843d-4967-a6b8-4be90d58b124 admin admin] PATH_INFO: `/v3/projects/8d1245be9be24a5fbae54af071270e21/users/bcb8c8a6104c49a48a81858b80fd0a1d/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.753623 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f451bf16-0889-44a7-91d5-50447208701e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:54.754371 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f451bf16-0889-44a7-91d5-50447208701e admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.754865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f451bf16-0889-44a7-91d5-50447208701e admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.755258 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f451bf16-0889-44a7-91d5-50447208701e admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.756066 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f451bf16-0889-44a7-91d5-50447208701e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:54.757930 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f451bf16-0889-44a7-91d5-50447208701e admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:54.764909 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca02a24d-843d-4967-a6b8-4be90d58b124 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8d1245be9be24a5fbae54af071270e21, user_id=bcb8c8a6104c49a48a81858b80fd0a1d, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:54.767567 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca02a24d-843d-4967-a6b8-4be90d58b124 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:54.771834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 799/1595] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:19:54 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:19:54.773926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071f39ff-7be3-4b43-8bc0-2ab782f08321 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.774244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071f39ff-7be3-4b43-8bc0-2ab782f08321 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.774587 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071f39ff-7be3-4b43-8bc0-2ab782f08321 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.786813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 797/1596] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:19:54 2022] PUT /identity/v3/projects/8d1245be9be24a5fbae54af071270e21/users/bcb8c8a6104c49a48a81858b80fd0a1d/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:19:54.787583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca02a24d-843d-4967-a6b8-4be90d58b124 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:54.796599 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-071f39ff-7be3-4b43-8bc0-2ab782f08321 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:54.798074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-071f39ff-7be3-4b43-8bc0-2ab782f08321 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:54.869545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca02a24d-843d-4967-a6b8-4be90d58b124 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:54.875355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 800/1597] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:54.875932 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0a42bd-3d48-4f1c-a061-dc6f8f04d8d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:54.876189 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071f39ff-7be3-4b43-8bc0-2ab782f08321 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:54.876900 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0a42bd-3d48-4f1c-a061-dc6f8f04d8d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.877182 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0a42bd-3d48-4f1c-a061-dc6f8f04d8d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.877457 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0a42bd-3d48-4f1c-a061-dc6f8f04d8d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.878457 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b0a42bd-3d48-4f1c-a061-dc6f8f04d8d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:54.879779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b0a42bd-3d48-4f1c-a061-dc6f8f04d8d9 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:54.887381 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071f39ff-7be3-4b43-8bc0-2ab782f08321 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:54.888282 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:19:54.893255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cfc823-a280-4548-8b3a-2c0b51ce964a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:54.894002 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cfc823-a280-4548-8b3a-2c0b51ce964a admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.894374 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cfc823-a280-4548-8b3a-2c0b51ce964a admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.894738 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cfc823-a280-4548-8b3a-2c0b51ce964a admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.895670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97cfc823-a280-4548-8b3a-2c0b51ce964a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:54.896097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 798/1598] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:19:54 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:19:54.897449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97cfc823-a280-4548-8b3a-2c0b51ce964a admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:54.898493 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcfd309-c64e-4a38-b098-167bc5314b53 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.898955 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcfd309-c64e-4a38-b098-167bc5314b53 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.899552 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcfd309-c64e-4a38-b098-167bc5314b53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.911228 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 801/1599] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:19:54 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:19:54.911228 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cfc823-a280-4548-8b3a-2c0b51ce964a admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:54.924439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cfc823-a280-4548-8b3a-2c0b51ce964a admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:54.924587 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-4bcfd309-c64e-4a38-b098-167bc5314b53 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:54.925700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-4bcfd309-c64e-4a38-b098-167bc5314b53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:54.930801 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e74dc7f-ce8d-4ef8-9d65-f66b63bc04b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:54.931831 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e74dc7f-ce8d-4ef8-9d65-f66b63bc04b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.932720 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e74dc7f-ce8d-4ef8-9d65-f66b63bc04b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.932720 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e74dc7f-ce8d-4ef8-9d65-f66b63bc04b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.933496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e74dc7f-ce8d-4ef8-9d65-f66b63bc04b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:54.935249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e74dc7f-ce8d-4ef8-9d65-f66b63bc04b7 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:54.937107 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 799/1600] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:54.938383 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcfd309-c64e-4a38-b098-167bc5314b53 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:54.948542 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcfd309-c64e-4a38-b098-167bc5314b53 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:54.950639 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 802/1601] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:19:54 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:19:54.952548 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb2a1d2-9470-4acb-ab8a-fc8242906716 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.955600 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb2a1d2-9470-4acb-ab8a-fc8242906716 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.955600 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb2a1d2-9470-4acb-ab8a-fc8242906716 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.955827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67840ad8-aaff-4d2d-93a5-68adbb553d00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:54.955827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67840ad8-aaff-4d2d-93a5-68adbb553d00 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.955827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67840ad8-aaff-4d2d-93a5-68adbb553d00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.956128 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67840ad8-aaff-4d2d-93a5-68adbb553d00 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:54.956701 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67840ad8-aaff-4d2d-93a5-68adbb553d00 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:54.958916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67840ad8-aaff-4d2d-93a5-68adbb553d00 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:54.974111 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 800/1602] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:19:54 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:19:54.974111 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67840ad8-aaff-4d2d-93a5-68adbb553d00 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:54.983905 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67840ad8-aaff-4d2d-93a5-68adbb553d00 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:54.987476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-dcb2a1d2-9470-4acb-ab8a-fc8242906716 None None] MFA Rules not processed for user `bcb8c8a6104c49a48a81858b80fd0a1d`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:54.990028 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-dcb2a1d2-9470-4acb-ab8a-fc8242906716 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:54.991219 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343cf007-9674-4924-8516-88ee9a3b405f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:54.991884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343cf007-9674-4924-8516-88ee9a3b405f admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:54.992192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343cf007-9674-4924-8516-88ee9a3b405f admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:54.992414 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343cf007-9674-4924-8516-88ee9a3b405f admin admin] PATH_INFO: `/v3/projects/0699af01ace347ecb12751b6b819e4de/users/f3434513afc649bdaaa398151d0a0a73/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:55.017454 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-343cf007-9674-4924-8516-88ee9a3b405f admin admin] RBAC: Authorizing `identity:create_grant(project_id=0699af01ace347ecb12751b6b819e4de, user_id=f3434513afc649bdaaa398151d0a0a73, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:55.021123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-343cf007-9674-4924-8516-88ee9a3b405f admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:55.055690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 801/1603] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:19:54 2022] PUT /identity/v3/projects/0699af01ace347ecb12751b6b819e4de/users/f3434513afc649bdaaa398151d0a0a73/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:19:55.055690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343cf007-9674-4924-8516-88ee9a3b405f admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:55.086490 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 803/1604] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:54 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:55.089827 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb2a1d2-9470-4acb-ab8a-fc8242906716 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:55.110322 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb2a1d2-9470-4acb-ab8a-fc8242906716 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:55.136125 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343cf007-9674-4924-8516-88ee9a3b405f admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:55.141688 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8ae642-f1a4-4178-a86d-ebac9d8ee5bb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:55.142683 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8ae642-f1a4-4178-a86d-ebac9d8ee5bb service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:55.143293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8ae642-f1a4-4178-a86d-ebac9d8ee5bb service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:55.143768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8ae642-f1a4-4178-a86d-ebac9d8ee5bb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:55.201348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a41b8cc-54c3-4b59-899f-b2df8a5f9c47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:55.202893 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a41b8cc-54c3-4b59-899f-b2df8a5f9c47 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:55.203886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a41b8cc-54c3-4b59-899f-b2df8a5f9c47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:55.204272 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a41b8cc-54c3-4b59-899f-b2df8a5f9c47 admin admin] PATH_INFO: `/v3/projects/7bdb5f97f83b400fb26c5bef99a23ecf/users/73d7523d0c2142dcb0c63ccacb5f6c10/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:55.209697 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:19:55.230934 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8ae642-f1a4-4178-a86d-ebac9d8ee5bb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:55.233087 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8ae642-f1a4-4178-a86d-ebac9d8ee5bb service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:55.235366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a41b8cc-54c3-4b59-899f-b2df8a5f9c47 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7bdb5f97f83b400fb26c5bef99a23ecf, user_id=73d7523d0c2142dcb0c63ccacb5f6c10, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:55.239262 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a41b8cc-54c3-4b59-899f-b2df8a5f9c47 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:55.246053 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 802/1605] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:19:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:19:55.246896 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8ae642-f1a4-4178-a86d-ebac9d8ee5bb service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:55.258609 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8ae642-f1a4-4178-a86d-ebac9d8ee5bb service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:55.274360 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 804/1606] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:19:55 2022] PUT /identity/v3/projects/7bdb5f97f83b400fb26c5bef99a23ecf/users/73d7523d0c2142dcb0c63ccacb5f6c10/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:19:55.274360 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a41b8cc-54c3-4b59-899f-b2df8a5f9c47 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:55.348241 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a41b8cc-54c3-4b59-899f-b2df8a5f9c47 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:55.353963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f463fad9-b555-46fb-b0be-515a7bb6b9d8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:55.354859 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f463fad9-b555-46fb-b0be-515a7bb6b9d8 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:55.355366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f463fad9-b555-46fb-b0be-515a7bb6b9d8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:55.355853 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f463fad9-b555-46fb-b0be-515a7bb6b9d8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:55.357952 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f66bc4e-9aad-4714-b9bb-f34a4284c826 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:55.358821 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f66bc4e-9aad-4714-b9bb-f34a4284c826 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:55.359163 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f66bc4e-9aad-4714-b9bb-f34a4284c826 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:55.359526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f66bc4e-9aad-4714-b9bb-f34a4284c826 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:55.360291 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f66bc4e-9aad-4714-b9bb-f34a4284c826 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:55.362298 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f66bc4e-9aad-4714-b9bb-f34a4284c826 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:55.369031 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:19:55.377354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f463fad9-b555-46fb-b0be-515a7bb6b9d8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:55.378748 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f463fad9-b555-46fb-b0be-515a7bb6b9d8 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:55.383319 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 803/1607] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:19:55 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:19:55.383319 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f66bc4e-9aad-4714-b9bb-f34a4284c826 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:55.392261 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 805/1608] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:19:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:19:55.393607 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ccfd43-1c91-4369-85ea-e5a5a4c0ebc3 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:55.393918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ccfd43-1c91-4369-85ea-e5a5a4c0ebc3 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:55.394179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ccfd43-1c91-4369-85ea-e5a5a4c0ebc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:55.397198 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f66bc4e-9aad-4714-b9bb-f34a4284c826 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:55.404465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3772b3-d62f-417d-af61-2cac547a581f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:55.404465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3772b3-d62f-417d-af61-2cac547a581f admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:55.404465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3772b3-d62f-417d-af61-2cac547a581f admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:55.404465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3772b3-d62f-417d-af61-2cac547a581f admin admin] PATH_INFO: `/v3/projects/52f37847ffde4c47a8716bff540866c1/users/0b31e8282eb843e59ad90e37882b8c2d/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:55.422173 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3772b3-d62f-417d-af61-2cac547a581f admin admin] RBAC: Authorizing `identity:create_grant(project_id=52f37847ffde4c47a8716bff540866c1, user_id=0b31e8282eb843e59ad90e37882b8c2d, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:55.425482 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3772b3-d62f-417d-af61-2cac547a581f admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:55.440132 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-d0ccfd43-1c91-4369-85ea-e5a5a4c0ebc3 None None] MFA Rules not processed for user `f3434513afc649bdaaa398151d0a0a73`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:55.441467 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-d0ccfd43-1c91-4369-85ea-e5a5a4c0ebc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:55.443902 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 804/1609] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:19:55 2022] PUT /identity/v3/projects/52f37847ffde4c47a8716bff540866c1/users/0b31e8282eb843e59ad90e37882b8c2d/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:19:55.445094 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3772b3-d62f-417d-af61-2cac547a581f admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:55.534203 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 806/1610] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:55 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:55.534400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3772b3-d62f-417d-af61-2cac547a581f admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:55.536737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcb8ddd-a562-4b91-ac62-a3cbcdd67044 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:55.537152 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcb8ddd-a562-4b91-ac62-a3cbcdd67044 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:55.537573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcb8ddd-a562-4b91-ac62-a3cbcdd67044 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:55.539382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e444a6b-83d8-4324-9f6e-1292d7c921a3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:55.540319 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e444a6b-83d8-4324-9f6e-1292d7c921a3 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:55.540598 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e444a6b-83d8-4324-9f6e-1292d7c921a3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:55.540867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e444a6b-83d8-4324-9f6e-1292d7c921a3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:55.595877 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-4bcb8ddd-a562-4b91-ac62-a3cbcdd67044 None None] MFA Rules not processed for user `73d7523d0c2142dcb0c63ccacb5f6c10`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:55.598919 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-4bcb8ddd-a562-4b91-ac62-a3cbcdd67044 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:55.626033 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e444a6b-83d8-4324-9f6e-1292d7c921a3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:55.628258 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e444a6b-83d8-4324-9f6e-1292d7c921a3 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:55.651767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 805/1611] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:19:55 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:19:55.652926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e444a6b-83d8-4324-9f6e-1292d7c921a3 service cinder] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:55.750107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 807/1612] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:55 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:55.752110 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48e6a66-2e19-4a5f-ac2a-92cea29420a4 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:55.752573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48e6a66-2e19-4a5f-ac2a-92cea29420a4 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:55.752991 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48e6a66-2e19-4a5f-ac2a-92cea29420a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:55.767616 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:19:55.789299 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-c48e6a66-2e19-4a5f-ac2a-92cea29420a4 None None] MFA Rules not processed for user `0b31e8282eb843e59ad90e37882b8c2d`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:55.789487 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e444a6b-83d8-4324-9f6e-1292d7c921a3 service cinder] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:55.791037 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c48e6a66-2e19-4a5f-ac2a-92cea29420a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:55.801297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aca1bc2-0f03-4dc0-a9f0-6e705070140f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:55.802023 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aca1bc2-0f03-4dc0-a9f0-6e705070140f admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:55.804392 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aca1bc2-0f03-4dc0-a9f0-6e705070140f admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:55.804692 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aca1bc2-0f03-4dc0-a9f0-6e705070140f admin admin] PATH_INFO: `/v3/projects/03e64281787145cc82cb14497b022a36/users/2524b556e9024894942bd7d4ed5a89e9/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:55.840486 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aca1bc2-0f03-4dc0-a9f0-6e705070140f admin admin] RBAC: Authorizing `identity:create_grant(project_id=03e64281787145cc82cb14497b022a36, user_id=2524b556e9024894942bd7d4ed5a89e9, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:55.844332 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aca1bc2-0f03-4dc0-a9f0-6e705070140f admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:55.872016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 806/1613] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:19:55 2022] PUT /identity/v3/projects/03e64281787145cc82cb14497b022a36/users/2524b556e9024894942bd7d4ed5a89e9/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:19:55.886125 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aca1bc2-0f03-4dc0-a9f0-6e705070140f admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:55.903930 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:19:55.907759 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 808/1614] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:55 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:55.908632 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48e6a66-2e19-4a5f-ac2a-92cea29420a4 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:55.939377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48e6a66-2e19-4a5f-ac2a-92cea29420a4 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:55.998800 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aca1bc2-0f03-4dc0-a9f0-6e705070140f admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:56.005514 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b071c4ce-1efa-4f21-801a-e28b8f7f9696 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:56.006284 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b071c4ce-1efa-4f21-801a-e28b8f7f9696 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:56.006659 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b071c4ce-1efa-4f21-801a-e28b8f7f9696 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:56.007032 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b071c4ce-1efa-4f21-801a-e28b8f7f9696 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:56.007884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b071c4ce-1efa-4f21-801a-e28b8f7f9696 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:56.010267 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b071c4ce-1efa-4f21-801a-e28b8f7f9696 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:56.052890 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 807/1615] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:19:55 2022] POST /identity/v3/projects => generated 363 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:19:56.054601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a65a68d-317a-4b5a-bc1b-f197bd1e0116 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:56.055079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a65a68d-317a-4b5a-bc1b-f197bd1e0116 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:56.055445 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a65a68d-317a-4b5a-bc1b-f197bd1e0116 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:56.064581 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90efafec-8924-4602-b018-0094c409ab99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:56.065722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90efafec-8924-4602-b018-0094c409ab99 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:56.066176 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90efafec-8924-4602-b018-0094c409ab99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:56.066756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90efafec-8924-4602-b018-0094c409ab99 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:56.067765 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90efafec-8924-4602-b018-0094c409ab99 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:56.070305 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90efafec-8924-4602-b018-0094c409ab99 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:56.101008 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-9a65a68d-317a-4b5a-bc1b-f197bd1e0116 None None] MFA Rules not processed for user `2524b556e9024894942bd7d4ed5a89e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:56.102698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-9a65a68d-317a-4b5a-bc1b-f197bd1e0116 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:56.122659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 809/1616] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:19:55 2022] POST /identity/v3/projects => generated 369 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:19:56.124673 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90efafec-8924-4602-b018-0094c409ab99 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:56.205082 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 808/1617] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:56 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:56.211844 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a65a68d-317a-4b5a-bc1b-f197bd1e0116 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:56.310020 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90efafec-8924-4602-b018-0094c409ab99 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:56.310020 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081d6b1c-fa2d-4301-9fb4-0c775e345cf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:56.310020 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081d6b1c-fa2d-4301-9fb4-0c775e345cf5 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:56.310020 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081d6b1c-fa2d-4301-9fb4-0c775e345cf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:56.310815 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081d6b1c-fa2d-4301-9fb4-0c775e345cf5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:56.310815 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081d6b1c-fa2d-4301-9fb4-0c775e345cf5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:56.310815 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081d6b1c-fa2d-4301-9fb4-0c775e345cf5 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:56.325671 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 810/1618] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:19:56 2022] POST /identity/v3/projects => generated 355 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:19:56.331661 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081d6b1c-fa2d-4301-9fb4-0c775e345cf5 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:56.351565 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a65a68d-317a-4b5a-bc1b-f197bd1e0116 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:56.361363 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:19:56.428692 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:19:56.455853 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081d6b1c-fa2d-4301-9fb4-0c775e345cf5 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:56.461622 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53025a8a-467f-4667-b0e1-5b295e6c32ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:56.462384 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53025a8a-467f-4667-b0e1-5b295e6c32ce admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:56.462723 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53025a8a-467f-4667-b0e1-5b295e6c32ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:56.463323 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53025a8a-467f-4667-b0e1-5b295e6c32ce admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:56.463692 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93fc5c6-69ae-4172-8fe6-6236f7eae674 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:56.464082 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53025a8a-467f-4667-b0e1-5b295e6c32ce admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:56.464940 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93fc5c6-69ae-4172-8fe6-6236f7eae674 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:56.465068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93fc5c6-69ae-4172-8fe6-6236f7eae674 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:56.465429 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93fc5c6-69ae-4172-8fe6-6236f7eae674 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:56.466162 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a93fc5c6-69ae-4172-8fe6-6236f7eae674 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:56.466908 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53025a8a-467f-4667-b0e1-5b295e6c32ce admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:56.469322 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a93fc5c6-69ae-4172-8fe6-6236f7eae674 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:56.531391 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 809/1619] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:19:56 2022] POST /identity/v3/users => generated 288 bytes in 326 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:19:56.532761 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93fc5c6-69ae-4172-8fe6-6236f7eae674 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:56.539229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 811/1620] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:19:56 2022] POST /identity/v3/users => generated 291 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:19:56.539229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53025a8a-467f-4667-b0e1-5b295e6c32ce admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:56.558723 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53025a8a-467f-4667-b0e1-5b295e6c32ce admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:56.561074 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93fc5c6-69ae-4172-8fe6-6236f7eae674 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:56.569163 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d81d477-a155-4d55-b8a7-3a09afbb056c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:56.573096 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62db905e-d19d-4c96-9621-71fdebc0a644 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:56.574481 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62db905e-d19d-4c96-9621-71fdebc0a644 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:56.574833 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62db905e-d19d-4c96-9621-71fdebc0a644 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:56.575177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62db905e-d19d-4c96-9621-71fdebc0a644 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:56.576127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d81d477-a155-4d55-b8a7-3a09afbb056c admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:56.578490 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62db905e-d19d-4c96-9621-71fdebc0a644 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:56.578916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d81d477-a155-4d55-b8a7-3a09afbb056c admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:56.579409 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d81d477-a155-4d55-b8a7-3a09afbb056c admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:56.580641 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62db905e-d19d-4c96-9621-71fdebc0a644 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:56.582540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d81d477-a155-4d55-b8a7-3a09afbb056c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:56.587421 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d81d477-a155-4d55-b8a7-3a09afbb056c admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:56.630436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 812/1621] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:19:56 2022] POST /identity/v3/users => generated 284 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:19:56.635841 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62db905e-d19d-4c96-9621-71fdebc0a644 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:56.650908 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62db905e-d19d-4c96-9621-71fdebc0a644 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:56.658972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f85bb043-f389-42b3-9b9f-3061614d5bc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:56.658972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85bb043-f389-42b3-9b9f-3061614d5bc8 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:56.658972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85bb043-f389-42b3-9b9f-3061614d5bc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:56.658972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85bb043-f389-42b3-9b9f-3061614d5bc8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:56.659380 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f85bb043-f389-42b3-9b9f-3061614d5bc8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:56.660057 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 810/1622] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:19:56 2022] POST /identity/v3/projects => generated 379 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:19:56.661529 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f85bb043-f389-42b3-9b9f-3061614d5bc8 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:56.663944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d81d477-a155-4d55-b8a7-3a09afbb056c admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:56.689377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 813/1623] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:19:56 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:19:56.690868 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f85bb043-f389-42b3-9b9f-3061614d5bc8 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:56.706902 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f85bb043-f389-42b3-9b9f-3061614d5bc8 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:56.782563 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d81d477-a155-4d55-b8a7-3a09afbb056c admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:56.790157 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a176a4a4-769e-4abb-8ba4-a4930a9cb3c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:56.791994 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a176a4a4-769e-4abb-8ba4-a4930a9cb3c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:56.792281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a176a4a4-769e-4abb-8ba4-a4930a9cb3c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:56.792728 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a176a4a4-769e-4abb-8ba4-a4930a9cb3c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:56.793474 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a176a4a4-769e-4abb-8ba4-a4930a9cb3c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:56.796372 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a176a4a4-769e-4abb-8ba4-a4930a9cb3c0 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:56.825778 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 811/1624] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:19:56 2022] GET /identity/v3/roles => generated 1420 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:19:56.828008 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a176a4a4-769e-4abb-8ba4-a4930a9cb3c0 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:56.842174 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a176a4a4-769e-4abb-8ba4-a4930a9cb3c0 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:56.844300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e86cfb-e6db-4b4a-8108-4fd2417e12a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:56.845146 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e86cfb-e6db-4b4a-8108-4fd2417e12a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:56.845343 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e86cfb-e6db-4b4a-8108-4fd2417e12a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:56.845646 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e86cfb-e6db-4b4a-8108-4fd2417e12a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:56.846434 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e86cfb-e6db-4b4a-8108-4fd2417e12a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:56.848427 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e86cfb-e6db-4b4a-8108-4fd2417e12a1 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:56.848858 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8387bc-c82e-4c86-b499-acbf0fe335f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:56.849464 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8387bc-c82e-4c86-b499-acbf0fe335f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:56.849763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8387bc-c82e-4c86-b499-acbf0fe335f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:56.850416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8387bc-c82e-4c86-b499-acbf0fe335f1 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:56.851211 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8387bc-c82e-4c86-b499-acbf0fe335f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:56.853654 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8387bc-c82e-4c86-b499-acbf0fe335f1 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:56.864479 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 814/1625] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:19:56 2022] GET /identity/v3/roles => generated 1420 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:19:56.864479 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e86cfb-e6db-4b4a-8108-4fd2417e12a1 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:56.880772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 812/1626] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:19:56 2022] POST /identity/v3/users => generated 296 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:19:56.881661 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8387bc-c82e-4c86-b499-acbf0fe335f1 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:56.887035 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e86cfb-e6db-4b4a-8108-4fd2417e12a1 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:56.897667 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8387bc-c82e-4c86-b499-acbf0fe335f1 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:56.907106 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c73302d-da2b-42ac-9272-a831e2ec4236 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:56.908027 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c73302d-da2b-42ac-9272-a831e2ec4236 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:56.908312 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c73302d-da2b-42ac-9272-a831e2ec4236 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:56.908633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c73302d-da2b-42ac-9272-a831e2ec4236 admin admin] PATH_INFO: `/v3/projects/7dede4c1da794d3db68881d477955d7d/users/c21a9f71b1814d22a5994eef312ae420/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:56.910590 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2ee239-76e0-4d8a-ac38-1af273f9b2eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:56.912030 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2ee239-76e0-4d8a-ac38-1af273f9b2eb admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:56.912324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2ee239-76e0-4d8a-ac38-1af273f9b2eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:56.912755 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2ee239-76e0-4d8a-ac38-1af273f9b2eb admin admin] PATH_INFO: `/v3/projects/df06e8751c89418db5fb916f63cac9d8/users/543a35f137b747acba1e665e60d6ac4d/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:56.933127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c73302d-da2b-42ac-9272-a831e2ec4236 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7dede4c1da794d3db68881d477955d7d, user_id=c21a9f71b1814d22a5994eef312ae420, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:56.936329 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c73302d-da2b-42ac-9272-a831e2ec4236 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:56.965628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2ee239-76e0-4d8a-ac38-1af273f9b2eb admin admin] RBAC: Authorizing `identity:create_grant(project_id=df06e8751c89418db5fb916f63cac9d8, user_id=543a35f137b747acba1e665e60d6ac4d, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:56.969096 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2ee239-76e0-4d8a-ac38-1af273f9b2eb admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:56.974750 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 813/1627] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:19:56 2022] PUT /identity/v3/projects/7dede4c1da794d3db68881d477955d7d/users/c21a9f71b1814d22a5994eef312ae420/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:19:56.976893 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c73302d-da2b-42ac-9272-a831e2ec4236 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:57.000427 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 815/1628] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:19:56 2022] PUT /identity/v3/projects/df06e8751c89418db5fb916f63cac9d8/users/543a35f137b747acba1e665e60d6ac4d/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:19:57.001394 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2ee239-76e0-4d8a-ac38-1af273f9b2eb admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:57.103011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:19:57.106460 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:19:57.107465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c73302d-da2b-42ac-9272-a831e2ec4236 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:57.161224 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2ee239-76e0-4d8a-ac38-1af273f9b2eb admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:57.171740 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d667e0-ce31-4545-b37e-1f27a5b10957 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:57.172384 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d667e0-ce31-4545-b37e-1f27a5b10957 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:57.172671 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d667e0-ce31-4545-b37e-1f27a5b10957 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:57.172950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d667e0-ce31-4545-b37e-1f27a5b10957 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:57.173613 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d667e0-ce31-4545-b37e-1f27a5b10957 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:57.179445 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d667e0-ce31-4545-b37e-1f27a5b10957 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:57.198729 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 816/1629] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:19:57 2022] GET /identity/v3/roles => generated 1420 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:19:57.199613 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d667e0-ce31-4545-b37e-1f27a5b10957 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:57.228095 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d667e0-ce31-4545-b37e-1f27a5b10957 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:57.233006 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9593628-c124-40b8-8d5c-9a894db58476 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:57.235274 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9593628-c124-40b8-8d5c-9a894db58476 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:57.235556 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9593628-c124-40b8-8d5c-9a894db58476 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:57.235916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9593628-c124-40b8-8d5c-9a894db58476 admin admin] PATH_INFO: `/v3/projects/d7f601e71ace4199be72b41bba1bbe83/users/6a15b1818fb442138bf9d479639978ce/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:57.245693 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c85125e-696c-4b30-bf3c-7b0c1608df42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:57.246369 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c85125e-696c-4b30-bf3c-7b0c1608df42 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:57.246817 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c85125e-696c-4b30-bf3c-7b0c1608df42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:57.247218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c85125e-696c-4b30-bf3c-7b0c1608df42 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:57.247888 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c85125e-696c-4b30-bf3c-7b0c1608df42 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:57.249889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c85125e-696c-4b30-bf3c-7b0c1608df42 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:57.260223 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9593628-c124-40b8-8d5c-9a894db58476 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d7f601e71ace4199be72b41bba1bbe83, user_id=6a15b1818fb442138bf9d479639978ce, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:57.266673 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9593628-c124-40b8-8d5c-9a894db58476 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:57.268340 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 817/1630] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:19:57 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:19:57.269284 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c85125e-696c-4b30-bf3c-7b0c1608df42 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:57.292512 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c85125e-696c-4b30-bf3c-7b0c1608df42 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:57.300863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 814/1631] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:19:56 2022] PUT /identity/v3/projects/d7f601e71ace4199be72b41bba1bbe83/users/6a15b1818fb442138bf9d479639978ce/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 325 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:19:57.302440 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9593628-c124-40b8-8d5c-9a894db58476 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:57.414419 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9593628-c124-40b8-8d5c-9a894db58476 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:57.430318 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366f8fab-1bce-4e7d-9333-ff209618447b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:57.430946 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366f8fab-1bce-4e7d-9333-ff209618447b admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:57.431230 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366f8fab-1bce-4e7d-9333-ff209618447b admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:57.431509 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366f8fab-1bce-4e7d-9333-ff209618447b admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:57.432170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366f8fab-1bce-4e7d-9333-ff209618447b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:57.433856 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366f8fab-1bce-4e7d-9333-ff209618447b admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:57.440717 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1582a19-9678-4e46-a01e-8103b1933101 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:57.441608 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1582a19-9678-4e46-a01e-8103b1933101 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:57.441931 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1582a19-9678-4e46-a01e-8103b1933101 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:57.442206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1582a19-9678-4e46-a01e-8103b1933101 admin admin] PATH_INFO: `/v3/projects/7bafa7cf29ca4c1cb3b8f660ef47b739/users/e3bbda34b8214ff9a5c60f99d7d07f4e/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:57.465449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 818/1632] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:19:57 2022] GET /identity/v3/roles => generated 1420 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:19:57.466511 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366f8fab-1bce-4e7d-9333-ff209618447b admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:57.468908 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1582a19-9678-4e46-a01e-8103b1933101 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7bafa7cf29ca4c1cb3b8f660ef47b739, user_id=e3bbda34b8214ff9a5c60f99d7d07f4e, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:57.484540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1582a19-9678-4e46-a01e-8103b1933101 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:57.491690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366f8fab-1bce-4e7d-9333-ff209618447b admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:57.500115 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338cfdc0-efa4-4a2d-a2d9-70dc85e18007 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:57.500791 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338cfdc0-efa4-4a2d-a2d9-70dc85e18007 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:57.501067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338cfdc0-efa4-4a2d-a2d9-70dc85e18007 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:57.501334 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338cfdc0-efa4-4a2d-a2d9-70dc85e18007 admin admin] PATH_INFO: `/v3/projects/7dede4c1da794d3db68881d477955d7d/users/c21a9f71b1814d22a5994eef312ae420/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:57.509338 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338cfdc0-efa4-4a2d-a2d9-70dc85e18007 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7dede4c1da794d3db68881d477955d7d, user_id=c21a9f71b1814d22a5994eef312ae420, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:57.510024 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 815/1633] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:19:57 2022] PUT /identity/v3/projects/7bafa7cf29ca4c1cb3b8f660ef47b739/users/e3bbda34b8214ff9a5c60f99d7d07f4e/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:19:57.511336 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338cfdc0-efa4-4a2d-a2d9-70dc85e18007 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:57.511667 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1582a19-9678-4e46-a01e-8103b1933101 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:57.549819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 819/1634] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:19:57 2022] PUT /identity/v3/projects/7dede4c1da794d3db68881d477955d7d/users/c21a9f71b1814d22a5994eef312ae420/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:19:57.549819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338cfdc0-efa4-4a2d-a2d9-70dc85e18007 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:57.645622 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1582a19-9678-4e46-a01e-8103b1933101 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:57.661845 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eff2e3e-3c97-40c2-acee-a1cf8c9529c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:57.662778 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eff2e3e-3c97-40c2-acee-a1cf8c9529c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:57.663146 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eff2e3e-3c97-40c2-acee-a1cf8c9529c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:57.663511 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eff2e3e-3c97-40c2-acee-a1cf8c9529c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:57.664043 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338cfdc0-efa4-4a2d-a2d9-70dc85e18007 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:57.665135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eff2e3e-3c97-40c2-acee-a1cf8c9529c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:57.667391 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eff2e3e-3c97-40c2-acee-a1cf8c9529c4 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:57.674249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032a7a73-3a72-4854-9ce8-016ed6a2f4bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:57.676570 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032a7a73-3a72-4854-9ce8-016ed6a2f4bd admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:57.677263 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032a7a73-3a72-4854-9ce8-016ed6a2f4bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:57.679073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032a7a73-3a72-4854-9ce8-016ed6a2f4bd admin admin] PATH_INFO: `/v3/projects/df06e8751c89418db5fb916f63cac9d8/users/543a35f137b747acba1e665e60d6ac4d/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:57.683669 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032a7a73-3a72-4854-9ce8-016ed6a2f4bd admin admin] RBAC: Authorizing `identity:create_grant(project_id=df06e8751c89418db5fb916f63cac9d8, user_id=543a35f137b747acba1e665e60d6ac4d, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:57.693152 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032a7a73-3a72-4854-9ce8-016ed6a2f4bd admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:57.696086 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 816/1635] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:19:57 2022] GET /identity/v3/roles => generated 1420 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:19:57.700305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eff2e3e-3c97-40c2-acee-a1cf8c9529c4 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:57.731117 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eff2e3e-3c97-40c2-acee-a1cf8c9529c4 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:57.731511 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 820/1636] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:19:57 2022] PUT /identity/v3/projects/df06e8751c89418db5fb916f63cac9d8/users/543a35f137b747acba1e665e60d6ac4d/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:19:57.733008 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590306b5-d397-4486-b259-46e1608cf06a None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:57.733824 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590306b5-d397-4486-b259-46e1608cf06a None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:57.734551 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590306b5-d397-4486-b259-46e1608cf06a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:57.741774 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:19:57.801297 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-590306b5-d397-4486-b259-46e1608cf06a None None] MFA Rules not processed for user `c21a9f71b1814d22a5994eef312ae420`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:57.802963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-590306b5-d397-4486-b259-46e1608cf06a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:57.858257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8cc4fe-0676-4e5d-a02b-2939d7ed6987 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:57.858903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8cc4fe-0676-4e5d-a02b-2939d7ed6987 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:57.859179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8cc4fe-0676-4e5d-a02b-2939d7ed6987 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:57.859448 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8cc4fe-0676-4e5d-a02b-2939d7ed6987 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:57.860089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8cc4fe-0676-4e5d-a02b-2939d7ed6987 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:57.862802 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8cc4fe-0676-4e5d-a02b-2939d7ed6987 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:57.871603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:19:57.887096 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 817/1637] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:19:57 2022] GET /identity/v3/roles => generated 1420 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:19:57.890490 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8cc4fe-0676-4e5d-a02b-2939d7ed6987 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:57.910242 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8cc4fe-0676-4e5d-a02b-2939d7ed6987 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:57.921517 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15714f8e-a444-4b0d-9b10-0076331c6319 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:57.923272 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15714f8e-a444-4b0d-9b10-0076331c6319 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:57.924063 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15714f8e-a444-4b0d-9b10-0076331c6319 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:57.925528 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15714f8e-a444-4b0d-9b10-0076331c6319 admin admin] PATH_INFO: `/v3/projects/d7f601e71ace4199be72b41bba1bbe83/users/6a15b1818fb442138bf9d479639978ce/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:57.929135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15714f8e-a444-4b0d-9b10-0076331c6319 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d7f601e71ace4199be72b41bba1bbe83, user_id=6a15b1818fb442138bf9d479639978ce, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:57.933235 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15714f8e-a444-4b0d-9b10-0076331c6319 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:57.958983 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 821/1638] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:57 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:57.962732 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1157160c-84c4-4e96-8648-a6166be296f8 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:57.963158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1157160c-84c4-4e96-8648-a6166be296f8 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:57.963528 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1157160c-84c4-4e96-8648-a6166be296f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:57.966085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 818/1639] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:19:57 2022] PUT /identity/v3/projects/d7f601e71ace4199be72b41bba1bbe83/users/6a15b1818fb442138bf9d479639978ce/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:19:57.968622 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15714f8e-a444-4b0d-9b10-0076331c6319 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:58.033321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-1157160c-84c4-4e96-8648-a6166be296f8 None None] MFA Rules not processed for user `543a35f137b747acba1e665e60d6ac4d`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:58.035324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-1157160c-84c4-4e96-8648-a6166be296f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:58.138109 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15714f8e-a444-4b0d-9b10-0076331c6319 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:58.151154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aba8a08-16d0-4216-a151-1fc6c4612a89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:58.152137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aba8a08-16d0-4216-a151-1fc6c4612a89 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:58.152417 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aba8a08-16d0-4216-a151-1fc6c4612a89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:58.152693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aba8a08-16d0-4216-a151-1fc6c4612a89 admin admin] PATH_INFO: `/v3/projects/7bafa7cf29ca4c1cb3b8f660ef47b739/users/e3bbda34b8214ff9a5c60f99d7d07f4e/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:58.157731 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aba8a08-16d0-4216-a151-1fc6c4612a89 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7bafa7cf29ca4c1cb3b8f660ef47b739, user_id=e3bbda34b8214ff9a5c60f99d7d07f4e, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:58.165492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aba8a08-16d0-4216-a151-1fc6c4612a89 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:58.199747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 819/1640] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:19:57 2022] PUT /identity/v3/projects/7bafa7cf29ca4c1cb3b8f660ef47b739/users/e3bbda34b8214ff9a5c60f99d7d07f4e/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:19:58.200789 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aba8a08-16d0-4216-a151-1fc6c4612a89 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:58.232769 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 822/1641] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:57 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:58.234753 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55263f02-768b-4899-8633-fc9f2cc780c0 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:58.235041 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55263f02-768b-4899-8633-fc9f2cc780c0 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:58.235314 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55263f02-768b-4899-8633-fc9f2cc780c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:58.287249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aba8a08-16d0-4216-a151-1fc6c4612a89 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:58.291838 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-55263f02-768b-4899-8633-fc9f2cc780c0 None None] MFA Rules not processed for user `6a15b1818fb442138bf9d479639978ce`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:58.292146 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9de1f01-aefe-4545-901a-0cc5f8f2823c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:58.292773 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9de1f01-aefe-4545-901a-0cc5f8f2823c service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:58.293035 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-55263f02-768b-4899-8633-fc9f2cc780c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:58.293172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9de1f01-aefe-4545-901a-0cc5f8f2823c service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:58.293353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9de1f01-aefe-4545-901a-0cc5f8f2823c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:58.352195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:19:58.387617 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9de1f01-aefe-4545-901a-0cc5f8f2823c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:58.389061 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9de1f01-aefe-4545-901a-0cc5f8f2823c service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:58.402584 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 820/1642] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:19:58 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:19:58.403732 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38f634a-ac30-4632-9817-a05e96964478 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:58.403998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38f634a-ac30-4632-9817-a05e96964478 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:58.404420 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38f634a-ac30-4632-9817-a05e96964478 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:58.435358 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-a38f634a-ac30-4632-9817-a05e96964478 None None] MFA Rules not processed for user `e3bbda34b8214ff9a5c60f99d7d07f4e`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:58.437281 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 823/1643] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:58 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:58.437896 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a38f634a-ac30-4632-9817-a05e96964478 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:58.438278 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55263f02-768b-4899-8633-fc9f2cc780c0 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:58.461342 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55263f02-768b-4899-8633-fc9f2cc780c0 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:58.476165 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385d5281-20e4-499b-8546-99158262a992 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:58.476867 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385d5281-20e4-499b-8546-99158262a992 service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:58.477149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385d5281-20e4-499b-8546-99158262a992 service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:58.477453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385d5281-20e4-499b-8546-99158262a992 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:58.530708 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:19:58.589201 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-385d5281-20e4-499b-8546-99158262a992 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:58.590684 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-385d5281-20e4-499b-8546-99158262a992 service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:58.609422 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 821/1644] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:19:58 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:58.610266 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefc16a8-89ef-4257-9db0-d3e6225afc24 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:58.610266 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefc16a8-89ef-4257-9db0-d3e6225afc24 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:58.610472 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefc16a8-89ef-4257-9db0-d3e6225afc24 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:58.614274 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 822/1645] 10.222.0.26 () {54 vars in 961 bytes} [Tue Sep 27 04:19:58 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:19:58.618413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefc16a8-89ef-4257-9db0-d3e6225afc24 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:58.619275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 824/1646] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:19:58 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:19:58.621248 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e09ea89-67c0-4fbe-a642-78a6b4e10918 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:58.622851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e09ea89-67c0-4fbe-a642-78a6b4e10918 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:58.624140 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e09ea89-67c0-4fbe-a642-78a6b4e10918 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:58.646460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefc16a8-89ef-4257-9db0-d3e6225afc24 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:58.658450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea0b4d7-05cb-4651-8169-afc1ac9ccbcb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:58.658865 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea0b4d7-05cb-4651-8169-afc1ac9ccbcb service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:58.659147 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea0b4d7-05cb-4651-8169-afc1ac9ccbcb service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:58.659490 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea0b4d7-05cb-4651-8169-afc1ac9ccbcb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:58.687186 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea0b4d7-05cb-4651-8169-afc1ac9ccbcb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:58.689084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea0b4d7-05cb-4651-8169-afc1ac9ccbcb service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:58.715480 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 823/1647] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:19:58 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:19:58.721324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-7e09ea89-67c0-4fbe-a642-78a6b4e10918 None None] MFA Rules not processed for user `a1586ee841cd44658e4d5feb25de7940`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:58.721680 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-7e09ea89-67c0-4fbe-a642-78a6b4e10918 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:58.770707 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea0b4d7-05cb-4651-8169-afc1ac9ccbcb service cinder] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:58.784001 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea0b4d7-05cb-4651-8169-afc1ac9ccbcb service cinder] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:58.796361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0e1fbd-6e44-429d-a4e7-b12774a76352 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:58.796983 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0e1fbd-6e44-429d-a4e7-b12774a76352 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:58.797261 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0e1fbd-6e44-429d-a4e7-b12774a76352 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:58.797857 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0e1fbd-6e44-429d-a4e7-b12774a76352 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:58.822575 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0e1fbd-6e44-429d-a4e7-b12774a76352 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:58.828481 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0e1fbd-6e44-429d-a4e7-b12774a76352 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:58.851943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 824/1648] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:19:58 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:19:58.891416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6ddfe5-2629-4a43-8886-2b5eba42bfab None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:58.893473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6ddfe5-2629-4a43-8886-2b5eba42bfab None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:58.894250 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6ddfe5-2629-4a43-8886-2b5eba42bfab None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:58.896820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 825/1650] 10.222.0.26 () {54 vars in 912 bytes} [Tue Sep 27 04:19:58 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:19:58.903942 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba544873-29bc-425e-ac16-9dddf962990c None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:58.906077 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba544873-29bc-425e-ac16-9dddf962990c None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:58.907519 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba544873-29bc-425e-ac16-9dddf962990c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:58.909888 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 825/1650] 10.222.0.26 () {58 vars in 1058 bytes} [Tue Sep 27 04:19:58 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 265 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:58.935570 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-ba544873-29bc-425e-ac16-9dddf962990c None None] MFA Rules not processed for user `a1586ee841cd44658e4d5feb25de7940`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:19:58.937523 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ba544873-29bc-425e-ac16-9dddf962990c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:19:58.951060 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 826/1651] 10.222.0.26 () {58 vars in 1009 bytes} [Tue Sep 27 04:19:58 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:19:59.001097 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e09ea89-67c0-4fbe-a642-78a6b4e10918 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:59.015333 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e09ea89-67c0-4fbe-a642-78a6b4e10918 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:59.029152 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5dabc2d-b2ae-4ddf-9a1b-5fefd040742f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:59.029985 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dabc2d-b2ae-4ddf-9a1b-5fefd040742f service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:59.031595 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dabc2d-b2ae-4ddf-9a1b-5fefd040742f service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:59.032110 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dabc2d-b2ae-4ddf-9a1b-5fefd040742f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:59.107089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba544873-29bc-425e-ac16-9dddf962990c None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:59.120039 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba544873-29bc-425e-ac16-9dddf962990c None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:59.135041 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a0d121-9869-4bb3-9b3d-21fbabe06337 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:59.138203 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a0d121-9869-4bb3-9b3d-21fbabe06337 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:59.138913 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a0d121-9869-4bb3-9b3d-21fbabe06337 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:59.139191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a0d121-9869-4bb3-9b3d-21fbabe06337 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:59.183195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a0d121-9869-4bb3-9b3d-21fbabe06337 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:59.185146 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a0d121-9869-4bb3-9b3d-21fbabe06337 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:59.209415 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5dabc2d-b2ae-4ddf-9a1b-5fefd040742f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:59.211000 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5dabc2d-b2ae-4ddf-9a1b-5fefd040742f service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:59.224380 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 827/1652] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:19:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:19:59.235218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 826/1653] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:19:58 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:19:59.397653 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a0d121-9869-4bb3-9b3d-21fbabe06337 service cinder] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:59.424440 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a0d121-9869-4bb3-9b3d-21fbabe06337 service cinder] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:59.434537 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d7aed8-a9eb-4d93-8ccb-0384ada67f62 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:59.444486 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d7aed8-a9eb-4d93-8ccb-0384ada67f62 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:59.445009 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d7aed8-a9eb-4d93-8ccb-0384ada67f62 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:59.445636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d7aed8-a9eb-4d93-8ccb-0384ada67f62 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:59.543045 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d7aed8-a9eb-4d93-8ccb-0384ada67f62 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:59.545259 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d7aed8-a9eb-4d93-8ccb-0384ada67f62 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:59.560057 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 828/1654] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:19:59 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:19:59.649918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5dabc2d-b2ae-4ddf-9a1b-5fefd040742f service cinder] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:19:59.649918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5dabc2d-b2ae-4ddf-9a1b-5fefd040742f service cinder] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:19:59.649918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee355391-c6d0-480f-83c7-ce456ac35503 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:19:59.649918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee355391-c6d0-480f-83c7-ce456ac35503 service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:19:59.651280 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee355391-c6d0-480f-83c7-ce456ac35503 service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:19:59.651280 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee355391-c6d0-480f-83c7-ce456ac35503 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:19:59.737192 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:19:59.763850 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee355391-c6d0-480f-83c7-ce456ac35503 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:19:59.766931 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee355391-c6d0-480f-83c7-ce456ac35503 service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:19:59.790565 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 827/1655] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:19:59 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:20:01.939492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ae0061-3071-47a1-8200-215c4a060e4f None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:01.940599 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ae0061-3071-47a1-8200-215c4a060e4f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:01.941075 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ae0061-3071-47a1-8200-215c4a060e4f None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:01.948434 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 829/1656] 10.222.0.26 () {54 vars in 961 bytes} [Tue Sep 27 04:20:01 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:20:01.971178 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceae0e67-50b2-4441-99dd-a6d723604343 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:01.977542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceae0e67-50b2-4441-99dd-a6d723604343 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:01.977882 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceae0e67-50b2-4441-99dd-a6d723604343 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:02.036137 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-ceae0e67-50b2-4441-99dd-a6d723604343 None None] MFA Rules not processed for user `a1586ee841cd44658e4d5feb25de7940`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:02.041365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ceae0e67-50b2-4441-99dd-a6d723604343 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:02.077229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 828/1657] 10.222.0.26 () {58 vars in 1058 bytes} [Tue Sep 27 04:20:01 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (3 switches on core 0) Sep 27 04:20:02.083308 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac00a4a-546b-44de-b1ac-3b175ea27fc4 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:02.084053 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac00a4a-546b-44de-b1ac-3b175ea27fc4 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:02.084576 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac00a4a-546b-44de-b1ac-3b175ea27fc4 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:02.086071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 830/1658] 10.222.0.26 () {54 vars in 912 bytes} [Tue Sep 27 04:20:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:20:02.092113 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e6b7c1-8842-4b06-b9cf-039f0dde08ef None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:02.092544 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e6b7c1-8842-4b06-b9cf-039f0dde08ef None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:02.093494 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e6b7c1-8842-4b06-b9cf-039f0dde08ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:02.153251 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-b1e6b7c1-8842-4b06-b9cf-039f0dde08ef None None] MFA Rules not processed for user `a1586ee841cd44658e4d5feb25de7940`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:02.154522 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-b1e6b7c1-8842-4b06-b9cf-039f0dde08ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:02.189366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 831/1659] 10.222.0.26 () {58 vars in 1009 bytes} [Tue Sep 27 04:20:02 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:20:02.251594 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceae0e67-50b2-4441-99dd-a6d723604343 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:02.264731 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceae0e67-50b2-4441-99dd-a6d723604343 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:02.276326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97abb596-78f7-4f81-9aa6-2769817b34d5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:02.277884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97abb596-78f7-4f81-9aa6-2769817b34d5 service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:02.278453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97abb596-78f7-4f81-9aa6-2769817b34d5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:02.278921 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97abb596-78f7-4f81-9aa6-2769817b34d5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:02.304302 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97abb596-78f7-4f81-9aa6-2769817b34d5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:02.306685 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97abb596-78f7-4f81-9aa6-2769817b34d5 service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:02.331649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 829/1660] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:02 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:04.843303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b9da2e-2d87-4072-b0b4-65ee56d1be21 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:04.843827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b9da2e-2d87-4072-b0b4-65ee56d1be21 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:04.843827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b9da2e-2d87-4072-b0b4-65ee56d1be21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:04.852930 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:04.913014 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-25b9da2e-2d87-4072-b0b4-65ee56d1be21 None None] MFA Rules not processed for user `f3438d850ce541cb9027ea3d5fe7dbf3`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:04.914151 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-25b9da2e-2d87-4072-b0b4-65ee56d1be21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:05.125877 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 832/1661] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:04 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 283 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:05.142598 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97abb596-78f7-4f81-9aa6-2769817b34d5 service cinder] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:05.154727 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97abb596-78f7-4f81-9aa6-2769817b34d5 service cinder] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:05.178402 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c9990e-abd6-4803-8bb5-06175492ac71 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:05.179026 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c9990e-abd6-4803-8bb5-06175492ac71 service nova] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:05.179312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c9990e-abd6-4803-8bb5-06175492ac71 service nova] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:05.179590 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c9990e-abd6-4803-8bb5-06175492ac71 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:05.190976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c9990e-abd6-4803-8bb5-06175492ac71 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:05.192569 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c9990e-abd6-4803-8bb5-06175492ac71 service nova] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:05.203851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 830/1662] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:05 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:05.414229 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d623f2ca-0cfe-46d8-9b45-8cbf2ae72e3c None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:05.415022 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d623f2ca-0cfe-46d8-9b45-8cbf2ae72e3c None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:05.415551 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d623f2ca-0cfe-46d8-9b45-8cbf2ae72e3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:05.488813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-d623f2ca-0cfe-46d8-9b45-8cbf2ae72e3c None None] MFA Rules not processed for user `03912e6e8a074eb1814c90158c246281`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:05.492572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d623f2ca-0cfe-46d8-9b45-8cbf2ae72e3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:05.688996 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 833/1663] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:05 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 276 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:05.689798 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b45855-d65b-41c8-bd1e-eb98b455fa9f None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:05.690169 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b45855-d65b-41c8-bd1e-eb98b455fa9f None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:05.690586 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b45855-d65b-41c8-bd1e-eb98b455fa9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:05.742231 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-01b45855-d65b-41c8-bd1e-eb98b455fa9f None None] MFA Rules not processed for user `03912e6e8a074eb1814c90158c246281`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:05.746538 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-01b45855-d65b-41c8-bd1e-eb98b455fa9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:05.775237 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 831/1664] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:05 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:20:05.795116 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d623f2ca-0cfe-46d8-9b45-8cbf2ae72e3c None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:05.821666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d623f2ca-0cfe-46d8-9b45-8cbf2ae72e3c None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:05.858055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43cea0b8-de38-4764-a24b-e3b6d3bc0adb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:05.861852 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cea0b8-de38-4764-a24b-e3b6d3bc0adb service nova] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:05.862410 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cea0b8-de38-4764-a24b-e3b6d3bc0adb service nova] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:05.864888 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cea0b8-de38-4764-a24b-e3b6d3bc0adb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:05.899723 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43cea0b8-de38-4764-a24b-e3b6d3bc0adb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:05.901268 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43cea0b8-de38-4764-a24b-e3b6d3bc0adb service nova] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:05.928107 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 834/1665] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:05 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:06.696989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6860612f-ca57-47e4-85a5-401cc7442c85 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:06.697612 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6860612f-ca57-47e4-85a5-401cc7442c85 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:06.697612 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6860612f-ca57-47e4-85a5-401cc7442c85 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:06.699384 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 832/1666] 10.222.0.26 () {54 vars in 912 bytes} [Tue Sep 27 04:20:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:20:06.706205 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96e8239-4a67-46c6-834d-4e5672b10cea None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:06.706490 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96e8239-4a67-46c6-834d-4e5672b10cea None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:06.706768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96e8239-4a67-46c6-834d-4e5672b10cea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:06.796877 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-a96e8239-4a67-46c6-834d-4e5672b10cea None None] MFA Rules not processed for user `cf29a8da103b411fbc79004eb371c05d`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:06.801494 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a96e8239-4a67-46c6-834d-4e5672b10cea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:06.957135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 835/1667] 10.222.0.26 () {58 vars in 1009 bytes} [Tue Sep 27 04:20:06 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:07.018147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6860612f-ca57-47e4-85a5-401cc7442c85 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:07.256076 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6860612f-ca57-47e4-85a5-401cc7442c85 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:07.263641 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130bbdd5-d474-486c-839d-1c0aa497a679 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:07.265100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130bbdd5-d474-486c-839d-1c0aa497a679 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:07.265708 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130bbdd5-d474-486c-839d-1c0aa497a679 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:07.266185 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130bbdd5-d474-486c-839d-1c0aa497a679 admin admin] PATH_INFO: `/v3/users/73d7523d0c2142dcb0c63ccacb5f6c10` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:07.283961 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130bbdd5-d474-486c-839d-1c0aa497a679 admin admin] RBAC: Authorizing `identity:delete_user(user_id=73d7523d0c2142dcb0c63ccacb5f6c10)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:07.291640 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130bbdd5-d474-486c-839d-1c0aa497a679 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:07.403335 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96e8239-4a67-46c6-834d-4e5672b10cea None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:07.422268 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-130bbdd5-d474-486c-839d-1c0aa497a679 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '73d7523d0c2142dcb0c63ccacb5f6c10'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:07.425567 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96e8239-4a67-46c6-834d-4e5672b10cea None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:07.459967 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbd44d6-3847-46ff-b30a-c47a562ad161 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:07.461248 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbd44d6-3847-46ff-b30a-c47a562ad161 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:07.461694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbd44d6-3847-46ff-b30a-c47a562ad161 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:07.462281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbd44d6-3847-46ff-b30a-c47a562ad161 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:07.467963 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:07.468932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-130bbdd5-d474-486c-839d-1c0aa497a679 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '73d7523d0c2142dcb0c63ccacb5f6c10'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:07.470385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-130bbdd5-d474-486c-839d-1c0aa497a679 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '73d7523d0c2142dcb0c63ccacb5f6c10'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:07.475065 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfbd44d6-3847-46ff-b30a-c47a562ad161 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:07.478893 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfbd44d6-3847-46ff-b30a-c47a562ad161 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:07.496225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 836/1668] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:07 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:07.499757 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-130bbdd5-d474-486c-839d-1c0aa497a679 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '73d7523d0c2142dcb0c63ccacb5f6c10'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:07.526472 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-130bbdd5-d474-486c-839d-1c0aa497a679 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '73d7523d0c2142dcb0c63ccacb5f6c10'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:07.543816 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 833/1669] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:07 2022] DELETE /identity/v3/users/73d7523d0c2142dcb0c63ccacb5f6c10 => generated 0 bytes in 523 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:07.547139 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd4a630-0198-42f5-ad0c-eedf893fd1f7 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:07.547449 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd4a630-0198-42f5-ad0c-eedf893fd1f7 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:07.547722 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd4a630-0198-42f5-ad0c-eedf893fd1f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:07.608100 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-bfd4a630-0198-42f5-ad0c-eedf893fd1f7 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:07.610015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-bfd4a630-0198-42f5-ad0c-eedf893fd1f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:07.777111 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 837/1670] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:07.872572 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130bbdd5-d474-486c-839d-1c0aa497a679 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:07.875778 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-130bbdd5-d474-486c-839d-1c0aa497a679 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:07.951221 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:08.044416 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130bbdd5-d474-486c-839d-1c0aa497a679 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:08.060635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2baca6a-1492-4812-aed0-d1e868feb2c5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:08.061336 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2baca6a-1492-4812-aed0-d1e868feb2c5 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:08.062398 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2baca6a-1492-4812-aed0-d1e868feb2c5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:08.067667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2baca6a-1492-4812-aed0-d1e868feb2c5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:08.102803 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2baca6a-1492-4812-aed0-d1e868feb2c5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:08.104252 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2baca6a-1492-4812-aed0-d1e868feb2c5 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:08.133505 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 834/1671] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 261 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:08.952071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c51089-bbdf-4a41-8a93-cbb9ac9f8ecf None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:08.953054 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c51089-bbdf-4a41-8a93-cbb9ac9f8ecf None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:08.953121 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c51089-bbdf-4a41-8a93-cbb9ac9f8ecf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:09.003242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2baca6a-1492-4812-aed0-d1e868feb2c5 service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:09.005473 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-b2baca6a-1492-4812-aed0-d1e868feb2c5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:09.034396 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-d3c51089-bbdf-4a41-8a93-cbb9ac9f8ecf None None] MFA Rules not processed for user `7c56e7333ce4450b8cbd5caeafc73947`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:09.036547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2baca6a-1492-4812-aed0-d1e868feb2c5 service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:09.037098 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d3c51089-bbdf-4a41-8a93-cbb9ac9f8ecf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:09.059183 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc02a86-d644-45e3-a1cf-37afdd7e1c5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:09.059904 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc02a86-d644-45e3-a1cf-37afdd7e1c5c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:09.060312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc02a86-d644-45e3-a1cf-37afdd7e1c5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:09.060682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc02a86-d644-45e3-a1cf-37afdd7e1c5c admin admin] PATH_INFO: `/v3/projects/7bdb5f97f83b400fb26c5bef99a23ecf` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:09.063241 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc02a86-d644-45e3-a1cf-37afdd7e1c5c admin admin] RBAC: Authorizing `identity:delete_project(project_id=7bdb5f97f83b400fb26c5bef99a23ecf)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:09.067612 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: WARNING py.warnings [None req-7cc02a86-d644-45e3-a1cf-37afdd7e1c5c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 27 04:20:09.067612 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: warnings.warn(msg) Sep 27 04:20:09.067612 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: Sep 27 04:20:09.070168 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc02a86-d644-45e3-a1cf-37afdd7e1c5c admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:09.194263 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 838/1672] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:08 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:09.206113 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c51089-bbdf-4a41-8a93-cbb9ac9f8ecf None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:09.211040 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d3c51089-bbdf-4a41-8a93-cbb9ac9f8ecf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:09.268790 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-7cc02a86-d644-45e3-a1cf-37afdd7e1c5c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7bdb5f97f83b400fb26c5bef99a23ecf', 'request_id': 'req-7cc02a86-d644-45e3-a1cf-37afdd7e1c5c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:09.293672 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-7cc02a86-d644-45e3-a1cf-37afdd7e1c5c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7bdb5f97f83b400fb26c5bef99a23ecf', 'request_id': 'req-7cc02a86-d644-45e3-a1cf-37afdd7e1c5c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:09.386878 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-7cc02a86-d644-45e3-a1cf-37afdd7e1c5c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7bdb5f97f83b400fb26c5bef99a23ecf', 'request_id': 'req-7cc02a86-d644-45e3-a1cf-37afdd7e1c5c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:09.399330 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c51089-bbdf-4a41-8a93-cbb9ac9f8ecf None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:09.453634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-7cc02a86-d644-45e3-a1cf-37afdd7e1c5c admin admin] The token cache is being invalidate because project 7bdb5f97f83b400fb26c5bef99a23ecf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:09.454296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-7cc02a86-d644-45e3-a1cf-37afdd7e1c5c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:09.456390 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-7cc02a86-d644-45e3-a1cf-37afdd7e1c5c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:09.466224 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 835/1673] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:09 2022] DELETE /identity/v3/projects/7bdb5f97f83b400fb26c5bef99a23ecf => generated 0 bytes in 459 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:09.467489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc02a86-d644-45e3-a1cf-37afdd7e1c5c admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:09.472453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-7cc02a86-d644-45e3-a1cf-37afdd7e1c5c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:09.495170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:09.616678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc02a86-d644-45e3-a1cf-37afdd7e1c5c admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:09.634832 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4395849-ec05-4a5d-896a-b80bd06994c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:09.635495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4395849-ec05-4a5d-896a-b80bd06994c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:09.635777 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4395849-ec05-4a5d-896a-b80bd06994c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:09.636052 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4395849-ec05-4a5d-896a-b80bd06994c4 admin admin] PATH_INFO: `/v3/users/c21a9f71b1814d22a5994eef312ae420` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:09.637510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4395849-ec05-4a5d-896a-b80bd06994c4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c21a9f71b1814d22a5994eef312ae420)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:09.639728 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4395849-ec05-4a5d-896a-b80bd06994c4 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:09.664674 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d74efd-59cf-4826-b92f-625e13eff1bc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:09.665665 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d74efd-59cf-4826-b92f-625e13eff1bc service nova] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:09.666192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d74efd-59cf-4826-b92f-625e13eff1bc service nova] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:09.666675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d74efd-59cf-4826-b92f-625e13eff1bc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:09.668738 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-54d74efd-59cf-4826-b92f-625e13eff1bc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:09.747722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-d4395849-ec05-4a5d-896a-b80bd06994c4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c21a9f71b1814d22a5994eef312ae420'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:09.764234 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:09.768902 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-d4395849-ec05-4a5d-896a-b80bd06994c4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c21a9f71b1814d22a5994eef312ae420'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:09.769926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-d4395849-ec05-4a5d-896a-b80bd06994c4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c21a9f71b1814d22a5994eef312ae420'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:09.782270 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d74efd-59cf-4826-b92f-625e13eff1bc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:09.784236 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d74efd-59cf-4826-b92f-625e13eff1bc service nova] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:09.788257 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-d4395849-ec05-4a5d-896a-b80bd06994c4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c21a9f71b1814d22a5994eef312ae420'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:09.801815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 839/1674] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:09 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 596 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:09.802876 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d74efd-59cf-4826-b92f-625e13eff1bc service nova] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:09.805523 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-54d74efd-59cf-4826-b92f-625e13eff1bc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:09.818976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-d4395849-ec05-4a5d-896a-b80bd06994c4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c21a9f71b1814d22a5994eef312ae420'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:09.833146 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 836/1675] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:09 2022] DELETE /identity/v3/users/c21a9f71b1814d22a5994eef312ae420 => generated 0 bytes in 363 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:09.948057 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d74efd-59cf-4826-b92f-625e13eff1bc service nova] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:10.088850 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:10.118834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f2f98b-4a3f-4164-8488-0a5235b669f3 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3a1bc198dfc0486e9e56d4fb28cda05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:10.119625 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f2f98b-4a3f-4164-8488-0a5235b669f3 service placement] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:10.121458 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f2f98b-4a3f-4164-8488-0a5235b669f3 service placement] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:10.123718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f2f98b-4a3f-4164-8488-0a5235b669f3 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:10.128819 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-94f2f98b-4a3f-4164-8488-0a5235b669f3 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:10.141266 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cd8389-4e29-460b-b7f9-1626015cf545 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:10.142682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cd8389-4e29-460b-b7f9-1626015cf545 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:10.143186 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cd8389-4e29-460b-b7f9-1626015cf545 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:10.160847 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f2f98b-4a3f-4164-8488-0a5235b669f3 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:10.162608 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f2f98b-4a3f-4164-8488-0a5235b669f3 service placement] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:10.191063 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 840/1676] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:09 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:10.207351 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-f2cd8389-4e29-460b-b7f9-1626015cf545 None None] MFA Rules not processed for user `f3438d850ce541cb9027ea3d5fe7dbf3`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:10.209401 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-f2cd8389-4e29-460b-b7f9-1626015cf545 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:10.369373 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 837/1677] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:10 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:10.390915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f2f98b-4a3f-4164-8488-0a5235b669f3 service placement] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:10.392725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-94f2f98b-4a3f-4164-8488-0a5235b669f3 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:10.620290 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f2f98b-4a3f-4164-8488-0a5235b669f3 service placement] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:10.627594 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4d5880-6067-4507-a96e-0527cd15b982 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:10.628505 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4d5880-6067-4507-a96e-0527cd15b982 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:10.628974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4d5880-6067-4507-a96e-0527cd15b982 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:10.629264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4d5880-6067-4507-a96e-0527cd15b982 admin admin] PATH_INFO: `/v3/projects/7dede4c1da794d3db68881d477955d7d` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:10.630634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4d5880-6067-4507-a96e-0527cd15b982 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7dede4c1da794d3db68881d477955d7d)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:10.634313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4d5880-6067-4507-a96e-0527cd15b982 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:10.835060 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-ab4d5880-6067-4507-a96e-0527cd15b982 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7dede4c1da794d3db68881d477955d7d', 'request_id': 'req-ab4d5880-6067-4507-a96e-0527cd15b982'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:10.895983 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-ab4d5880-6067-4507-a96e-0527cd15b982 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7dede4c1da794d3db68881d477955d7d', 'request_id': 'req-ab4d5880-6067-4507-a96e-0527cd15b982'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:10.909727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-ab4d5880-6067-4507-a96e-0527cd15b982 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7dede4c1da794d3db68881d477955d7d', 'request_id': 'req-ab4d5880-6067-4507-a96e-0527cd15b982'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:10.930426 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-ab4d5880-6067-4507-a96e-0527cd15b982 admin admin] The token cache is being invalidate because project 7dede4c1da794d3db68881d477955d7d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:10.930763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-ab4d5880-6067-4507-a96e-0527cd15b982 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:10.931481 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-ab4d5880-6067-4507-a96e-0527cd15b982 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:10.933628 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 841/1678] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:10 2022] DELETE /identity/v3/projects/7dede4c1da794d3db68881d477955d7d => generated 0 bytes in 544 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:20:10.939358 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31243d1d-8674-4586-97c1-803cd75542e7 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:10.939661 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31243d1d-8674-4586-97c1-803cd75542e7 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:10.939949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31243d1d-8674-4586-97c1-803cd75542e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:11.003470 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-31243d1d-8674-4586-97c1-803cd75542e7 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:11.006133 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-31243d1d-8674-4586-97c1-803cd75542e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:11.226198 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 838/1679] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:11.228630 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beeded10-49ef-47d3-8741-bc387d041f99 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:11.229386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beeded10-49ef-47d3-8741-bc387d041f99 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:11.229820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beeded10-49ef-47d3-8741-bc387d041f99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:11.276811 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-beeded10-49ef-47d3-8741-bc387d041f99 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:11.279425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-beeded10-49ef-47d3-8741-bc387d041f99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:11.328526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 842/1680] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:11.330641 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31243d1d-8674-4586-97c1-803cd75542e7 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:11.353109 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31243d1d-8674-4586-97c1-803cd75542e7 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:11.359932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ceeeb21-a2d1-4496-a000-2b85638ce5be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:11.360483 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ceeeb21-a2d1-4496-a000-2b85638ce5be admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:11.360765 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ceeeb21-a2d1-4496-a000-2b85638ce5be admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:11.361040 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ceeeb21-a2d1-4496-a000-2b85638ce5be admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:11.364626 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ceeeb21-a2d1-4496-a000-2b85638ce5be admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:20:11.365001 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ceeeb21-a2d1-4496-a000-2b85638ce5be admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:11.370468 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ceeeb21-a2d1-4496-a000-2b85638ce5be admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:11.447070 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 839/1681] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:20:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:20:11.464216 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beeded10-49ef-47d3-8741-bc387d041f99 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:11.481995 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beeded10-49ef-47d3-8741-bc387d041f99 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:11.496921 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249d2061-fda4-4318-b607-b6a375bfe222 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:11.497311 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249d2061-fda4-4318-b607-b6a375bfe222 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:11.498540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249d2061-fda4-4318-b607-b6a375bfe222 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:11.498540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249d2061-fda4-4318-b607-b6a375bfe222 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:11.499812 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249d2061-fda4-4318-b607-b6a375bfe222 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:11.506045 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249d2061-fda4-4318-b607-b6a375bfe222 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:11.581216 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 843/1682] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:11 2022] POST /identity/v3/projects => generated 387 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:11.585408 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ceeeb21-a2d1-4496-a000-2b85638ce5be admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:11.604336 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ceeeb21-a2d1-4496-a000-2b85638ce5be admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:11.903513 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3d4a19-aa28-4af6-a3df-3d7b17fbcc95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:11.904176 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3d4a19-aa28-4af6-a3df-3d7b17fbcc95 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:11.904460 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3d4a19-aa28-4af6-a3df-3d7b17fbcc95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:11.904738 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3d4a19-aa28-4af6-a3df-3d7b17fbcc95 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:11.908307 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3d4a19-aa28-4af6-a3df-3d7b17fbcc95 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:11.915511 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3d4a19-aa28-4af6-a3df-3d7b17fbcc95 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:11.952723 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=94,keystone:INSERT=3,keystone:DELETE=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:12.004953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 840/1683] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:11 2022] POST /identity/v3/users => generated 294 bytes in 418 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:12.005800 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249d2061-fda4-4318-b607-b6a375bfe222 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:12.023242 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249d2061-fda4-4318-b607-b6a375bfe222 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:12.042203 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e91536-c1ca-4ede-a36e-787d66307085 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:12.042845 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e91536-c1ca-4ede-a36e-787d66307085 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:12.043176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e91536-c1ca-4ede-a36e-787d66307085 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:12.043470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e91536-c1ca-4ede-a36e-787d66307085 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:12.044169 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e91536-c1ca-4ede-a36e-787d66307085 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:12.055795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e91536-c1ca-4ede-a36e-787d66307085 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:12.117383 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 844/1684] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:12 2022] GET /identity/v3/roles => generated 1420 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:12.118908 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3d4a19-aa28-4af6-a3df-3d7b17fbcc95 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:12.163715 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3d4a19-aa28-4af6-a3df-3d7b17fbcc95 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:12.198410 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1a1fbf-95e2-4db5-8be5-5cf66ee2edbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:12.201630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1a1fbf-95e2-4db5-8be5-5cf66ee2edbd admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:12.204224 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1a1fbf-95e2-4db5-8be5-5cf66ee2edbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:12.208135 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1a1fbf-95e2-4db5-8be5-5cf66ee2edbd admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:12.211300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1a1fbf-95e2-4db5-8be5-5cf66ee2edbd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:12.212861 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1a1fbf-95e2-4db5-8be5-5cf66ee2edbd admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:12.257787 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e91536-c1ca-4ede-a36e-787d66307085 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:12.260787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 841/1685] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:12 2022] GET /identity/v3/roles => generated 1420 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:20:12.286919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e91536-c1ca-4ede-a36e-787d66307085 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:12.309709 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cc122b-a405-4aa2-83ed-a093027f8f31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:12.313891 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cc122b-a405-4aa2-83ed-a093027f8f31 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:12.315901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cc122b-a405-4aa2-83ed-a093027f8f31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:12.318508 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cc122b-a405-4aa2-83ed-a093027f8f31 admin admin] PATH_INFO: `/v3/projects/1abc25ba289d49f285f776cd65e3e3ea/users/2f5d29f9f1814716a43ec5dcfeb5c747/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:12.407205 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cc122b-a405-4aa2-83ed-a093027f8f31 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1abc25ba289d49f285f776cd65e3e3ea, user_id=2f5d29f9f1814716a43ec5dcfeb5c747, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:12.410364 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cc122b-a405-4aa2-83ed-a093027f8f31 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:12.443013 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 845/1686] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:12 2022] PUT /identity/v3/projects/1abc25ba289d49f285f776cd65e3e3ea/users/2f5d29f9f1814716a43ec5dcfeb5c747/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:20:12.454813 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2f3a95-46c5-4cc1-86e8-a80b97acaffe None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:12.455109 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2f3a95-46c5-4cc1-86e8-a80b97acaffe None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:12.455381 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2f3a95-46c5-4cc1-86e8-a80b97acaffe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:12.572539 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-7f2f3a95-46c5-4cc1-86e8-a80b97acaffe None None] MFA Rules not processed for user `2f5d29f9f1814716a43ec5dcfeb5c747`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:12.577405 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-7f2f3a95-46c5-4cc1-86e8-a80b97acaffe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:12.743786 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 842/1687] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:12 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 291 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:12.749481 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cc122b-a405-4aa2-83ed-a093027f8f31 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:12.803170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:12.954004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cc122b-a405-4aa2-83ed-a093027f8f31 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:12.964874 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8701d8a4-b5bc-4567-83f1-cfd5349bfca7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:12.966154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8701d8a4-b5bc-4567-83f1-cfd5349bfca7 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:12.966653 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8701d8a4-b5bc-4567-83f1-cfd5349bfca7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:12.969262 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8701d8a4-b5bc-4567-83f1-cfd5349bfca7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:12.970952 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8701d8a4-b5bc-4567-83f1-cfd5349bfca7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:12.981297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8701d8a4-b5bc-4567-83f1-cfd5349bfca7 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:13.051167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 846/1688] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:12 2022] POST /identity/v3/projects => generated 387 bytes in 301 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:13.051547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2f3a95-46c5-4cc1-86e8-a80b97acaffe None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:13.082614 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2f3a95-46c5-4cc1-86e8-a80b97acaffe None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:13.304730 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8701d8a4-b5bc-4567-83f1-cfd5349bfca7 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:13.307874 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-8701d8a4-b5bc-4567-83f1-cfd5349bfca7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:13.363185 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9975bf4d-6b85-4d0a-9638-2241dda09cf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:13.363789 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9975bf4d-6b85-4d0a-9638-2241dda09cf2 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:13.364081 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9975bf4d-6b85-4d0a-9638-2241dda09cf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:13.364357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9975bf4d-6b85-4d0a-9638-2241dda09cf2 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:13.365005 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9975bf4d-6b85-4d0a-9638-2241dda09cf2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:13.382945 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9975bf4d-6b85-4d0a-9638-2241dda09cf2 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:13.460582 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 843/1689] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:13 2022] POST /identity/v3/users => generated 294 bytes in 409 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:20:13.462045 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9975bf4d-6b85-4d0a-9638-2241dda09cf2 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:13.492250 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9975bf4d-6b85-4d0a-9638-2241dda09cf2 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:13.505869 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8701d8a4-b5bc-4567-83f1-cfd5349bfca7 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:13.519834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6add0056-f82a-4465-9185-8823eb7ea004 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:13.521665 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2f554d-e363-4ce9-b174-2f3314673374 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:13.521846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6add0056-f82a-4465-9185-8823eb7ea004 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:13.522114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6add0056-f82a-4465-9185-8823eb7ea004 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:13.522392 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6add0056-f82a-4465-9185-8823eb7ea004 admin admin] PATH_INFO: `/v3/users/2524b556e9024894942bd7d4ed5a89e9` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:13.523021 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2f554d-e363-4ce9-b174-2f3314673374 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:13.523480 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2f554d-e363-4ce9-b174-2f3314673374 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:13.523931 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2f554d-e363-4ce9-b174-2f3314673374 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:13.524766 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce2f554d-e363-4ce9-b174-2f3314673374 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:13.526599 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce2f554d-e363-4ce9-b174-2f3314673374 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:13.528251 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6add0056-f82a-4465-9185-8823eb7ea004 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2524b556e9024894942bd7d4ed5a89e9)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:13.530317 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: WARNING py.warnings [None req-6add0056-f82a-4465-9185-8823eb7ea004 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 27 04:20:13.530317 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: warnings.warn(msg) Sep 27 04:20:13.530317 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: Sep 27 04:20:13.537510 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6add0056-f82a-4465-9185-8823eb7ea004 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:13.568048 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 844/1690] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:13 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:13.575645 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2f554d-e363-4ce9-b174-2f3314673374 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:13.608963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2f554d-e363-4ce9-b174-2f3314673374 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:13.621872 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811094c6-eede-4b74-b0c2-2d8a8e0fa122 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:13.622659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811094c6-eede-4b74-b0c2-2d8a8e0fa122 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:13.624467 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811094c6-eede-4b74-b0c2-2d8a8e0fa122 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:13.624959 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811094c6-eede-4b74-b0c2-2d8a8e0fa122 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:13.627032 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-811094c6-eede-4b74-b0c2-2d8a8e0fa122 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:13.630321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-811094c6-eede-4b74-b0c2-2d8a8e0fa122 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:13.709687 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 845/1691] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:13 2022] GET /identity/v3/roles => generated 1420 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:13.714559 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811094c6-eede-4b74-b0c2-2d8a8e0fa122 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:13.730241 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811094c6-eede-4b74-b0c2-2d8a8e0fa122 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:13.743003 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d0908c-202f-49ba-a87b-b26d819197e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:13.745297 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d0908c-202f-49ba-a87b-b26d819197e0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:13.746342 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d0908c-202f-49ba-a87b-b26d819197e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:13.747348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d0908c-202f-49ba-a87b-b26d819197e0 admin admin] PATH_INFO: `/v3/projects/dcfbd01e12e548719c49888950d8cb56/users/5827d0b3c9f64f5cacef8b76c3c550ec/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:13.781876 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d0908c-202f-49ba-a87b-b26d819197e0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dcfbd01e12e548719c49888950d8cb56, user_id=5827d0b3c9f64f5cacef8b76c3c550ec, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:13.790846 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d0908c-202f-49ba-a87b-b26d819197e0 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:13.805770 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6add0056-f82a-4465-9185-8823eb7ea004 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2524b556e9024894942bd7d4ed5a89e9'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:13.821975 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 846/1692] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:13 2022] PUT /identity/v3/projects/dcfbd01e12e548719c49888950d8cb56/users/5827d0b3c9f64f5cacef8b76c3c550ec/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:20:13.835743 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae119a0-9562-459a-851a-d231bb0147fc None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:13.836232 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae119a0-9562-459a-851a-d231bb0147fc None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:13.836925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae119a0-9562-459a-851a-d231bb0147fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:13.848223 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6add0056-f82a-4465-9185-8823eb7ea004 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2524b556e9024894942bd7d4ed5a89e9'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:13.850817 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6add0056-f82a-4465-9185-8823eb7ea004 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2524b556e9024894942bd7d4ed5a89e9'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:13.873019 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6add0056-f82a-4465-9185-8823eb7ea004 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2524b556e9024894942bd7d4ed5a89e9'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:13.925597 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6add0056-f82a-4465-9185-8823eb7ea004 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2524b556e9024894942bd7d4ed5a89e9'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:13.940261 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 847/1693] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:13 2022] DELETE /identity/v3/users/2524b556e9024894942bd7d4ed5a89e9 => generated 0 bytes in 636 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:13.944669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05fc79f-844c-4442-ae14-e19adafa6a82 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:13.944929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05fc79f-844c-4442-ae14-e19adafa6a82 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:13.945203 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05fc79f-844c-4442-ae14-e19adafa6a82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:13.990889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-dae119a0-9562-459a-851a-d231bb0147fc None None] MFA Rules not processed for user `5827d0b3c9f64f5cacef8b76c3c550ec`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:13.992309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-dae119a0-9562-459a-851a-d231bb0147fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:13.993303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-d05fc79f-844c-4442-ae14-e19adafa6a82 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:13.995964 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d05fc79f-844c-4442-ae14-e19adafa6a82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:14.011038 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:14.031299 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:14.162287 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 848/1694] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:14.181335 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d05fc79f-844c-4442-ae14-e19adafa6a82 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:14.182948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d05fc79f-844c-4442-ae14-e19adafa6a82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:14.240385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 847/1695] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:13 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 406 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:14.245020 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae119a0-9562-459a-851a-d231bb0147fc None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:14.250636 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-dae119a0-9562-459a-851a-d231bb0147fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:14.284377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae119a0-9562-459a-851a-d231bb0147fc None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:14.295860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed09ca3-574e-4099-bd4f-d4b7197672e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:14.296720 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed09ca3-574e-4099-bd4f-d4b7197672e0 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:14.309540 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed09ca3-574e-4099-bd4f-d4b7197672e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:14.312276 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed09ca3-574e-4099-bd4f-d4b7197672e0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:14.313301 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed09ca3-574e-4099-bd4f-d4b7197672e0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:14.317291 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed09ca3-574e-4099-bd4f-d4b7197672e0 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:14.387359 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 848/1696] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:14 2022] POST /identity/v3/projects => generated 387 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:14.390930 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13ddcf5-3b20-44eb-8055-41c8e775d1bc None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:14.391394 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13ddcf5-3b20-44eb-8055-41c8e775d1bc None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:14.391768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13ddcf5-3b20-44eb-8055-41c8e775d1bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:14.442681 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d05fc79f-844c-4442-ae14-e19adafa6a82 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:14.450794 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-d13ddcf5-3b20-44eb-8055-41c8e775d1bc None None] MFA Rules not processed for user `7c56e7333ce4450b8cbd5caeafc73947`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:14.452495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-d13ddcf5-3b20-44eb-8055-41c8e775d1bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:14.542256 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f63884e-af78-44e0-807b-66f0f6e59725 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:14.543286 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f63884e-af78-44e0-807b-66f0f6e59725 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:14.548956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f63884e-af78-44e0-807b-66f0f6e59725 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:14.549565 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f63884e-af78-44e0-807b-66f0f6e59725 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:14.656451 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 849/1697] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:14 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 268 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:14.657319 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13ddcf5-3b20-44eb-8055-41c8e775d1bc None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:14.718278 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f63884e-af78-44e0-807b-66f0f6e59725 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:14.719957 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f63884e-af78-44e0-807b-66f0f6e59725 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:14.751820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 849/1698] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 571 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:14.782885 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f63884e-af78-44e0-807b-66f0f6e59725 service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:14.784255 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-3f63884e-af78-44e0-807b-66f0f6e59725 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:14.826197 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f63884e-af78-44e0-807b-66f0f6e59725 service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:14.837511 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c1d931-5595-4367-a8c7-21bc684f3afb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:14.840190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c1d931-5595-4367-a8c7-21bc684f3afb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:14.840841 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c1d931-5595-4367-a8c7-21bc684f3afb admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:14.842217 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c1d931-5595-4367-a8c7-21bc684f3afb admin admin] PATH_INFO: `/v3/users/bcb8c8a6104c49a48a81858b80fd0a1d` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:14.845459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c1d931-5595-4367-a8c7-21bc684f3afb admin admin] RBAC: Authorizing `identity:delete_user(user_id=bcb8c8a6104c49a48a81858b80fd0a1d)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:14.851941 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13ddcf5-3b20-44eb-8055-41c8e775d1bc None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:14.852332 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c1d931-5595-4367-a8c7-21bc684f3afb admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:14.868283 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3347f085-4621-415f-b685-978bcebfe634 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:14.868922 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3347f085-4621-415f-b685-978bcebfe634 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:14.869242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3347f085-4621-415f-b685-978bcebfe634 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:14.869517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3347f085-4621-415f-b685-978bcebfe634 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:14.870336 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3347f085-4621-415f-b685-978bcebfe634 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:14.872313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3347f085-4621-415f-b685-978bcebfe634 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:14.916859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:14.972711 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 850/1699] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:14 2022] POST /identity/v3/users => generated 300 bytes in 315 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:14.975510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3347f085-4621-415f-b685-978bcebfe634 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:15.027881 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-30c1d931-5595-4367-a8c7-21bc684f3afb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bcb8c8a6104c49a48a81858b80fd0a1d'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:15.031019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3347f085-4621-415f-b685-978bcebfe634 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:15.040843 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00715f39-f9ea-4221-b013-1a6c26fdb985 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:15.041266 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00715f39-f9ea-4221-b013-1a6c26fdb985 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:15.041600 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00715f39-f9ea-4221-b013-1a6c26fdb985 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:15.042049 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00715f39-f9ea-4221-b013-1a6c26fdb985 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:15.042581 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00715f39-f9ea-4221-b013-1a6c26fdb985 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:15.044283 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00715f39-f9ea-4221-b013-1a6c26fdb985 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:15.057988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 851/1700] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:14 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:20:15.058391 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-30c1d931-5595-4367-a8c7-21bc684f3afb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bcb8c8a6104c49a48a81858b80fd0a1d'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:15.061522 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-30c1d931-5595-4367-a8c7-21bc684f3afb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bcb8c8a6104c49a48a81858b80fd0a1d'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:15.062855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00715f39-f9ea-4221-b013-1a6c26fdb985 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:15.064885 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-00715f39-f9ea-4221-b013-1a6c26fdb985 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:15.081400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-30c1d931-5595-4367-a8c7-21bc684f3afb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bcb8c8a6104c49a48a81858b80fd0a1d'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:15.083781 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00715f39-f9ea-4221-b013-1a6c26fdb985 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:15.092059 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99399b34-3de4-4f5e-bfa6-66d35c3653c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:15.093772 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99399b34-3de4-4f5e-bfa6-66d35c3653c8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:15.094408 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99399b34-3de4-4f5e-bfa6-66d35c3653c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:15.094942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99399b34-3de4-4f5e-bfa6-66d35c3653c8 admin admin] PATH_INFO: `/v3/projects/d4d4abd3268e446fb4a1867b2ab17227/users/da7a4bc732154f728255fbb3b7b7ec83/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:15.100420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:15.107448 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-30c1d931-5595-4367-a8c7-21bc684f3afb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bcb8c8a6104c49a48a81858b80fd0a1d'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:15.120645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 850/1701] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:14 2022] DELETE /identity/v3/users/bcb8c8a6104c49a48a81858b80fd0a1d => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:20:15.125870 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a1fbc0-0d70-4493-946f-b9bb9cb71d67 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:15.126081 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a1fbc0-0d70-4493-946f-b9bb9cb71d67 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:15.126316 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a1fbc0-0d70-4493-946f-b9bb9cb71d67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:15.151044 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99399b34-3de4-4f5e-bfa6-66d35c3653c8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d4d4abd3268e446fb4a1867b2ab17227, user_id=da7a4bc732154f728255fbb3b7b7ec83, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:15.166234 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99399b34-3de4-4f5e-bfa6-66d35c3653c8 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:15.181408 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-26a1fbc0-0d70-4493-946f-b9bb9cb71d67 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:15.184355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-26a1fbc0-0d70-4493-946f-b9bb9cb71d67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:15.251677 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 852/1702] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:15 2022] PUT /identity/v3/projects/d4d4abd3268e446fb4a1867b2ab17227/users/da7a4bc732154f728255fbb3b7b7ec83/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:15.256476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99399b34-3de4-4f5e-bfa6-66d35c3653c8 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:15.357726 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 851/1703] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:15.362252 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af84351c-e4d4-4c8f-ab71-83f69e42398a None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:15.363000 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af84351c-e4d4-4c8f-ab71-83f69e42398a None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:15.363300 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af84351c-e4d4-4c8f-ab71-83f69e42398a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:15.457332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99399b34-3de4-4f5e-bfa6-66d35c3653c8 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:15.491114 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f840e7e5-26af-415b-8a1e-10bb3c94ea0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:15.493012 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f840e7e5-26af-415b-8a1e-10bb3c94ea0d admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:15.493451 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f840e7e5-26af-415b-8a1e-10bb3c94ea0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:15.493916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f840e7e5-26af-415b-8a1e-10bb3c94ea0d admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:15.494782 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f840e7e5-26af-415b-8a1e-10bb3c94ea0d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:15.499687 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f840e7e5-26af-415b-8a1e-10bb3c94ea0d admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:15.526130 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 853/1704] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:15 2022] GET /identity/v3/roles => generated 1420 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:15.527380 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f840e7e5-26af-415b-8a1e-10bb3c94ea0d admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:15.541343 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-f840e7e5-26af-415b-8a1e-10bb3c94ea0d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:15.584844 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-af84351c-e4d4-4c8f-ab71-83f69e42398a None None] MFA Rules not processed for user `5a2fcc8c9ac44dceadcc30d163bbfa0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:15.592609 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-af84351c-e4d4-4c8f-ab71-83f69e42398a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:15.775375 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 852/1705] 10.222.0.26 () {58 vars in 1009 bytes} [Tue Sep 27 04:20:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 417 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:15.776328 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af84351c-e4d4-4c8f-ab71-83f69e42398a None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:15.804538 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af84351c-e4d4-4c8f-ab71-83f69e42398a None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:15.832685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac40eda-f9a7-4e53-b74d-d8a65f48d827 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:15.833382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac40eda-f9a7-4e53-b74d-d8a65f48d827 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:15.833665 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac40eda-f9a7-4e53-b74d-d8a65f48d827 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:15.834881 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac40eda-f9a7-4e53-b74d-d8a65f48d827 admin admin] PATH_INFO: `/v3/projects/d4d4abd3268e446fb4a1867b2ab17227/users/da7a4bc732154f728255fbb3b7b7ec83/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:15.839740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac40eda-f9a7-4e53-b74d-d8a65f48d827 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d4d4abd3268e446fb4a1867b2ab17227, user_id=da7a4bc732154f728255fbb3b7b7ec83, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:15.842444 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac40eda-f9a7-4e53-b74d-d8a65f48d827 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:15.850742 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f840e7e5-26af-415b-8a1e-10bb3c94ea0d admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:15.871622 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=92,keystone:DELETE=6,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:15.880513 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c4f964-4d9b-480d-b554-b89e618a6466 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:15.883264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c4f964-4d9b-480d-b554-b89e618a6466 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:15.884360 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c4f964-4d9b-480d-b554-b89e618a6466 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:15.884789 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c4f964-4d9b-480d-b554-b89e618a6466 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:15.900047 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 853/1706] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:15 2022] PUT /identity/v3/projects/d4d4abd3268e446fb4a1867b2ab17227/users/da7a4bc732154f728255fbb3b7b7ec83/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:15.907406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98efaacb-7958-478a-bad2-a9b15e57c4b1 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:15.909902 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98efaacb-7958-478a-bad2-a9b15e57c4b1 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:15.911589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98efaacb-7958-478a-bad2-a9b15e57c4b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:15.932637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c4f964-4d9b-480d-b554-b89e618a6466 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:15.936232 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c4f964-4d9b-480d-b554-b89e618a6466 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:15.969349 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 854/1707] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 443 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:15.991692 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-98efaacb-7958-478a-bad2-a9b15e57c4b1 None None] MFA Rules not processed for user `da7a4bc732154f728255fbb3b7b7ec83`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:15.994127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-98efaacb-7958-478a-bad2-a9b15e57c4b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:16.132879 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c4f964-4d9b-480d-b554-b89e618a6466 service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:16.135202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-43c4f964-4d9b-480d-b554-b89e618a6466 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:16.208257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 854/1708] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:15 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 307 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:16.359526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98efaacb-7958-478a-bad2-a9b15e57c4b1 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:16.362673 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-98efaacb-7958-478a-bad2-a9b15e57c4b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:16.557609 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98efaacb-7958-478a-bad2-a9b15e57c4b1 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:16.565435 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:16.576087 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1325cbfc-c836-4f67-b700-7f1f9035567d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:16.576955 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1325cbfc-c836-4f67-b700-7f1f9035567d service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:16.578028 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1325cbfc-c836-4f67-b700-7f1f9035567d service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:16.579124 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1325cbfc-c836-4f67-b700-7f1f9035567d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:16.587434 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c4f964-4d9b-480d-b554-b89e618a6466 service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:16.594659 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1325cbfc-c836-4f67-b700-7f1f9035567d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:16.599413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1325cbfc-c836-4f67-b700-7f1f9035567d service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:16.614831 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 855/1709] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:16 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 256 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:20:16.703237 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1325cbfc-c836-4f67-b700-7f1f9035567d service cinder] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:16.704839 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-1325cbfc-c836-4f67-b700-7f1f9035567d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:16.841211 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2087dd-cae9-4206-9c5e-76b6dea571f4 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '181dd5dc720c4a97860852062b7b19c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:16.844084 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2087dd-cae9-4206-9c5e-76b6dea571f4 service swift] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:16.844423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2087dd-cae9-4206-9c5e-76b6dea571f4 service swift] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:16.845346 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2087dd-cae9-4206-9c5e-76b6dea571f4 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:16.888078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1325cbfc-c836-4f67-b700-7f1f9035567d service cinder] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:16.904089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa09cbe-d881-4649-8e17-27dcd4eaff78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:16.904746 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa09cbe-d881-4649-8e17-27dcd4eaff78 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:16.905198 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa09cbe-d881-4649-8e17-27dcd4eaff78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:16.906475 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa09cbe-d881-4649-8e17-27dcd4eaff78 admin admin] PATH_INFO: `/v3/projects/03e64281787145cc82cb14497b022a36` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:16.909521 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa09cbe-d881-4649-8e17-27dcd4eaff78 admin admin] RBAC: Authorizing `identity:delete_project(project_id=03e64281787145cc82cb14497b022a36)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:16.914451 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa09cbe-d881-4649-8e17-27dcd4eaff78 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:16.995317 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:17.004886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2087dd-cae9-4206-9c5e-76b6dea571f4 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:17.010894 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2087dd-cae9-4206-9c5e-76b6dea571f4 service swift] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:17.025511 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-efa09cbe-d881-4649-8e17-27dcd4eaff78 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '03e64281787145cc82cb14497b022a36', 'request_id': 'req-efa09cbe-d881-4649-8e17-27dcd4eaff78'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:17.033017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 855/1710] 10.222.0.26 () {60 vars in 1384 bytes} [Tue Sep 27 04:20:16 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 901 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 04:20:17.034685 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2087dd-cae9-4206-9c5e-76b6dea571f4 service swift] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:17.036479 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-efa09cbe-d881-4649-8e17-27dcd4eaff78 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '03e64281787145cc82cb14497b022a36', 'request_id': 'req-efa09cbe-d881-4649-8e17-27dcd4eaff78'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:17.038192 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ad2087dd-cae9-4206-9c5e-76b6dea571f4 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:17.052626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-efa09cbe-d881-4649-8e17-27dcd4eaff78 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '03e64281787145cc82cb14497b022a36', 'request_id': 'req-efa09cbe-d881-4649-8e17-27dcd4eaff78'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:17.081087 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2087dd-cae9-4206-9c5e-76b6dea571f4 service swift] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:17.108229 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-efa09cbe-d881-4649-8e17-27dcd4eaff78 admin admin] The token cache is being invalidate because project 03e64281787145cc82cb14497b022a36 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:17.109937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-efa09cbe-d881-4649-8e17-27dcd4eaff78 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:17.115485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-efa09cbe-d881-4649-8e17-27dcd4eaff78 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:17.119770 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 856/1711] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:16 2022] DELETE /identity/v3/projects/03e64281787145cc82cb14497b022a36 => generated 0 bytes in 417 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:20:17.124411 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa09cbe-d881-4649-8e17-27dcd4eaff78 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:17.126989 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-efa09cbe-d881-4649-8e17-27dcd4eaff78 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:17.253154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706f4037-573b-4d02-acf0-818d89939b35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:17.254326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706f4037-573b-4d02-acf0-818d89939b35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:17.254854 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706f4037-573b-4d02-acf0-818d89939b35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:17.255267 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706f4037-573b-4d02-acf0-818d89939b35 admin admin] PATH_INFO: `/v3/users/f3434513afc649bdaaa398151d0a0a73` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:17.256871 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706f4037-573b-4d02-acf0-818d89939b35 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f3434513afc649bdaaa398151d0a0a73)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:17.259689 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706f4037-573b-4d02-acf0-818d89939b35 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:17.315110 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa09cbe-d881-4649-8e17-27dcd4eaff78 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:17.335498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c918adfc-a151-4811-9b08-d34cf437bb23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:17.336169 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c918adfc-a151-4811-9b08-d34cf437bb23 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:17.336453 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c918adfc-a151-4811-9b08-d34cf437bb23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:17.336727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c918adfc-a151-4811-9b08-d34cf437bb23 admin admin] PATH_INFO: `/v3/users/e3bbda34b8214ff9a5c60f99d7d07f4e` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:17.338150 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c918adfc-a151-4811-9b08-d34cf437bb23 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e3bbda34b8214ff9a5c60f99d7d07f4e)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:17.340461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c918adfc-a151-4811-9b08-d34cf437bb23 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:17.393094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-706f4037-573b-4d02-acf0-818d89939b35 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f3434513afc649bdaaa398151d0a0a73'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:17.395011 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:17.412244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-706f4037-573b-4d02-acf0-818d89939b35 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f3434513afc649bdaaa398151d0a0a73'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:17.413106 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-706f4037-573b-4d02-acf0-818d89939b35 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f3434513afc649bdaaa398151d0a0a73'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:17.425243 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-706f4037-573b-4d02-acf0-818d89939b35 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f3434513afc649bdaaa398151d0a0a73'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:17.453459 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-706f4037-573b-4d02-acf0-818d89939b35 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f3434513afc649bdaaa398151d0a0a73'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:17.475653 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 856/1712] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:17 2022] DELETE /identity/v3/users/f3434513afc649bdaaa398151d0a0a73 => generated 0 bytes in 442 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:17.483523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9f0393-06c8-460a-9964-6e8d22b10809 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:17.484275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9f0393-06c8-460a-9964-6e8d22b10809 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:17.491254 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9f0393-06c8-460a-9964-6e8d22b10809 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:17.494988 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-c918adfc-a151-4811-9b08-d34cf437bb23 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e3bbda34b8214ff9a5c60f99d7d07f4e'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:17.531116 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-c918adfc-a151-4811-9b08-d34cf437bb23 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e3bbda34b8214ff9a5c60f99d7d07f4e'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:17.534232 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-c918adfc-a151-4811-9b08-d34cf437bb23 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e3bbda34b8214ff9a5c60f99d7d07f4e'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:17.562141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-c918adfc-a151-4811-9b08-d34cf437bb23 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e3bbda34b8214ff9a5c60f99d7d07f4e'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:17.566079 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-bd9f0393-06c8-460a-9964-6e8d22b10809 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:17.568486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-bd9f0393-06c8-460a-9964-6e8d22b10809 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:17.601352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-c918adfc-a151-4811-9b08-d34cf437bb23 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e3bbda34b8214ff9a5c60f99d7d07f4e'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:17.635810 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 857/1713] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:17 2022] DELETE /identity/v3/users/e3bbda34b8214ff9a5c60f99d7d07f4e => generated 0 bytes in 510 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:17.818005 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 857/1714] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 333 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:17.845428 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c918adfc-a151-4811-9b08-d34cf437bb23 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:17.847884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c918adfc-a151-4811-9b08-d34cf437bb23 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:18.092578 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:18.164201 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c918adfc-a151-4811-9b08-d34cf437bb23 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:18.173706 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9f0393-06c8-460a-9964-6e8d22b10809 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:18.175518 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-bd9f0393-06c8-460a-9964-6e8d22b10809 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:18.177125 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335e5d61-a4e0-4a4c-b9f6-cc78d8996f25 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:18.180633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335e5d61-a4e0-4a4c-b9f6-cc78d8996f25 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:18.180980 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335e5d61-a4e0-4a4c-b9f6-cc78d8996f25 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:18.181263 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335e5d61-a4e0-4a4c-b9f6-cc78d8996f25 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:18.403354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9f0393-06c8-460a-9964-6e8d22b10809 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:18.473596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335e5d61-a4e0-4a4c-b9f6-cc78d8996f25 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:18.478466 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335e5d61-a4e0-4a4c-b9f6-cc78d8996f25 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:18.535412 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 858/1715] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 690 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:18.587292 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2faa1879-eac3-4679-9fd9-293405df1f87 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:18.588736 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2faa1879-eac3-4679-9fd9-293405df1f87 service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:18.590306 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2faa1879-eac3-4679-9fd9-293405df1f87 service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:18.603908 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2faa1879-eac3-4679-9fd9-293405df1f87 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:18.622454 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-2faa1879-eac3-4679-9fd9-293405df1f87 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:18.713896 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:18.780808 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2faa1879-eac3-4679-9fd9-293405df1f87 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:18.788287 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2faa1879-eac3-4679-9fd9-293405df1f87 service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:18.810389 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 858/1716] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:18 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 637 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:20:19.170664 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335e5d61-a4e0-4a4c-b9f6-cc78d8996f25 service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:19.173140 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-335e5d61-a4e0-4a4c-b9f6-cc78d8996f25 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:19.200923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335e5d61-a4e0-4a4c-b9f6-cc78d8996f25 service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:19.226529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3963e64-3f0a-46c6-aba8-febc8e0f4905 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:19.227200 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3963e64-3f0a-46c6-aba8-febc8e0f4905 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:19.227478 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3963e64-3f0a-46c6-aba8-febc8e0f4905 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:19.227755 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3963e64-3f0a-46c6-aba8-febc8e0f4905 admin admin] PATH_INFO: `/v3/projects/8d1245be9be24a5fbae54af071270e21` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:19.235155 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3963e64-3f0a-46c6-aba8-febc8e0f4905 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8d1245be9be24a5fbae54af071270e21)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:19.237256 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3963e64-3f0a-46c6-aba8-febc8e0f4905 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:19.401623 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-b3963e64-3f0a-46c6-aba8-febc8e0f4905 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8d1245be9be24a5fbae54af071270e21', 'request_id': 'req-b3963e64-3f0a-46c6-aba8-febc8e0f4905'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:19.446816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-b3963e64-3f0a-46c6-aba8-febc8e0f4905 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8d1245be9be24a5fbae54af071270e21', 'request_id': 'req-b3963e64-3f0a-46c6-aba8-febc8e0f4905'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:19.473650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-b3963e64-3f0a-46c6-aba8-febc8e0f4905 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8d1245be9be24a5fbae54af071270e21', 'request_id': 'req-b3963e64-3f0a-46c6-aba8-febc8e0f4905'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:19.511902 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-b3963e64-3f0a-46c6-aba8-febc8e0f4905 admin admin] The token cache is being invalidate because project 8d1245be9be24a5fbae54af071270e21 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:19.512620 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-b3963e64-3f0a-46c6-aba8-febc8e0f4905 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:19.514324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-b3963e64-3f0a-46c6-aba8-febc8e0f4905 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:19.517161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 859/1717] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:19 2022] DELETE /identity/v3/projects/8d1245be9be24a5fbae54af071270e21 => generated 0 bytes in 347 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:19.521999 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb34d495-9bb6-4557-9185-d68c4a90b793 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:19.522292 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb34d495-9bb6-4557-9185-d68c4a90b793 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:19.522565 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb34d495-9bb6-4557-9185-d68c4a90b793 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:19.545354 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3963e64-3f0a-46c6-aba8-febc8e0f4905 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:19.546782 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-b3963e64-3f0a-46c6-aba8-febc8e0f4905 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:19.572408 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-eb34d495-9bb6-4557-9185-d68c4a90b793 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:19.573647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-eb34d495-9bb6-4557-9185-d68c4a90b793 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:19.763508 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3963e64-3f0a-46c6-aba8-febc8e0f4905 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:19.776467 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbe5a20-6196-48ea-91f5-be3af0faa8e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:19.777855 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbe5a20-6196-48ea-91f5-be3af0faa8e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:19.778525 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbe5a20-6196-48ea-91f5-be3af0faa8e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:19.779097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbe5a20-6196-48ea-91f5-be3af0faa8e7 admin admin] PATH_INFO: `/v3/projects/7bafa7cf29ca4c1cb3b8f660ef47b739` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:19.781496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 859/1718] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:20:19.782688 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cbe5a20-6196-48ea-91f5-be3af0faa8e7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7bafa7cf29ca4c1cb3b8f660ef47b739)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:19.785189 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cbe5a20-6196-48ea-91f5-be3af0faa8e7 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:19.791743 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af6e087-6587-4e3b-84ee-4921d5082069 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:19.793035 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af6e087-6587-4e3b-84ee-4921d5082069 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:19.794953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af6e087-6587-4e3b-84ee-4921d5082069 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:19.839625 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:19.874669 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-8af6e087-6587-4e3b-84ee-4921d5082069 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:19.874669 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-8af6e087-6587-4e3b-84ee-4921d5082069 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:19.906069 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 860/1719] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:20:19.907279 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813ec092-3303-4c0b-afd1-4127a5f40d82 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:19.907568 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813ec092-3303-4c0b-afd1-4127a5f40d82 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:19.907841 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813ec092-3303-4c0b-afd1-4127a5f40d82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:19.944566 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0cbe5a20-6196-48ea-91f5-be3af0faa8e7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7bafa7cf29ca4c1cb3b8f660ef47b739', 'request_id': 'req-0cbe5a20-6196-48ea-91f5-be3af0faa8e7'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:19.955086 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-813ec092-3303-4c0b-afd1-4127a5f40d82 None None] MFA Rules not processed for user `cf29a8da103b411fbc79004eb371c05d`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:19.961521 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-813ec092-3303-4c0b-afd1-4127a5f40d82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:19.972721 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0cbe5a20-6196-48ea-91f5-be3af0faa8e7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7bafa7cf29ca4c1cb3b8f660ef47b739', 'request_id': 'req-0cbe5a20-6196-48ea-91f5-be3af0faa8e7'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:20.006018 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0cbe5a20-6196-48ea-91f5-be3af0faa8e7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7bafa7cf29ca4c1cb3b8f660ef47b739', 'request_id': 'req-0cbe5a20-6196-48ea-91f5-be3af0faa8e7'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:20.045487 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0cbe5a20-6196-48ea-91f5-be3af0faa8e7 admin admin] The token cache is being invalidate because project 7bafa7cf29ca4c1cb3b8f660ef47b739 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:20.046613 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 861/1720] 10.222.0.57 () {58 vars in 1015 bytes} [Tue Sep 27 04:20:19 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:20.047087 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0cbe5a20-6196-48ea-91f5-be3af0faa8e7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:20.047326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813ec092-3303-4c0b-afd1-4127a5f40d82 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:20.049690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0cbe5a20-6196-48ea-91f5-be3af0faa8e7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:20.055044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 860/1721] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:19 2022] DELETE /identity/v3/projects/7bafa7cf29ca4c1cb3b8f660ef47b739 => generated 0 bytes in 510 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:20.058489 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbe5a20-6196-48ea-91f5-be3af0faa8e7 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:20.063416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-0cbe5a20-6196-48ea-91f5-be3af0faa8e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:20.151461 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813ec092-3303-4c0b-afd1-4127a5f40d82 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:20.182971 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da3b9fb-aa62-45b7-826c-c76e4bb687ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:20.186187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da3b9fb-aa62-45b7-826c-c76e4bb687ae admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:20.189242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da3b9fb-aa62-45b7-826c-c76e4bb687ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:20.189696 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da3b9fb-aa62-45b7-826c-c76e4bb687ae admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:20.191538 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da3b9fb-aa62-45b7-826c-c76e4bb687ae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:20:20.192666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da3b9fb-aa62-45b7-826c-c76e4bb687ae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:20.195921 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da3b9fb-aa62-45b7-826c-c76e4bb687ae admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:20.228746 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 862/1722] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:20:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:20:20.257713 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3b27fb-bdce-49e6-94c7-3d44a493456f None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:20.258673 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3b27fb-bdce-49e6-94c7-3d44a493456f None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:20.260762 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3b27fb-bdce-49e6-94c7-3d44a493456f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:20.315113 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbe5a20-6196-48ea-91f5-be3af0faa8e7 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:20.321435 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-4b3b27fb-bdce-49e6-94c7-3d44a493456f None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:20.322788 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-4b3b27fb-bdce-49e6-94c7-3d44a493456f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:20.327260 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9040d007-39cc-4d4c-a748-45b42875f9ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:20.328754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9040d007-39cc-4d4c-a748-45b42875f9ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:20.329562 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9040d007-39cc-4d4c-a748-45b42875f9ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:20.330264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9040d007-39cc-4d4c-a748-45b42875f9ed admin admin] PATH_INFO: `/v3/projects/0699af01ace347ecb12751b6b819e4de` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:20.331525 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9040d007-39cc-4d4c-a748-45b42875f9ed admin admin] RBAC: Authorizing `identity:delete_project(project_id=0699af01ace347ecb12751b6b819e4de)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:20.333606 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9040d007-39cc-4d4c-a748-45b42875f9ed admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:20.338650 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 863/1723] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:20.339933 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3b27fb-bdce-49e6-94c7-3d44a493456f None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:20.362650 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-4b3b27fb-bdce-49e6-94c7-3d44a493456f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:20.425931 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3b27fb-bdce-49e6-94c7-3d44a493456f None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:20.440280 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:20.451873 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-9040d007-39cc-4d4c-a748-45b42875f9ed admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0699af01ace347ecb12751b6b819e4de', 'request_id': 'req-9040d007-39cc-4d4c-a748-45b42875f9ed'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:20.461441 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-9040d007-39cc-4d4c-a748-45b42875f9ed admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0699af01ace347ecb12751b6b819e4de', 'request_id': 'req-9040d007-39cc-4d4c-a748-45b42875f9ed'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:20.474228 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-9040d007-39cc-4d4c-a748-45b42875f9ed admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0699af01ace347ecb12751b6b819e4de', 'request_id': 'req-9040d007-39cc-4d4c-a748-45b42875f9ed'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:20.512961 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-9040d007-39cc-4d4c-a748-45b42875f9ed admin admin] The token cache is being invalidate because project 0699af01ace347ecb12751b6b819e4de was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:20.513256 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-9040d007-39cc-4d4c-a748-45b42875f9ed admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:20.518591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-9040d007-39cc-4d4c-a748-45b42875f9ed admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:20.523937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 861/1724] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:20 2022] DELETE /identity/v3/projects/0699af01ace347ecb12751b6b819e4de => generated 0 bytes in 467 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:20.540068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9040d007-39cc-4d4c-a748-45b42875f9ed admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:20.546043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-9040d007-39cc-4d4c-a748-45b42875f9ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:20.611848 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d5b541-b7a2-4863-a8b1-6ffc78e3a4bf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:20.613105 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d5b541-b7a2-4863-a8b1-6ffc78e3a4bf service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:20.613972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d5b541-b7a2-4863-a8b1-6ffc78e3a4bf service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:20.616107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d5b541-b7a2-4863-a8b1-6ffc78e3a4bf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:20.619378 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-29d5b541-b7a2-4863-a8b1-6ffc78e3a4bf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:20.653962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d5b541-b7a2-4863-a8b1-6ffc78e3a4bf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:20.662854 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d5b541-b7a2-4863-a8b1-6ffc78e3a4bf service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:20.671139 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:20.695659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 864/1725] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:20 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:20.700876 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c388f88-9134-4100-bca3-8b6138e65a89 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:20.703462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c388f88-9134-4100-bca3-8b6138e65a89 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:20.704354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c388f88-9134-4100-bca3-8b6138e65a89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:20.794641 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9040d007-39cc-4d4c-a748-45b42875f9ed admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:20.795734 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-7c388f88-9134-4100-bca3-8b6138e65a89 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:20.799262 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-7c388f88-9134-4100-bca3-8b6138e65a89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:20.810646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4686d52-d8fa-4028-8453-11781d7e54fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:20.812995 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4686d52-d8fa-4028-8453-11781d7e54fc admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:20.812995 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4686d52-d8fa-4028-8453-11781d7e54fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:20.814014 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4686d52-d8fa-4028-8453-11781d7e54fc admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:20.814642 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4686d52-d8fa-4028-8453-11781d7e54fc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:20.816608 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4686d52-d8fa-4028-8453-11781d7e54fc admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:20.828951 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 865/1726] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:20.834663 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c388f88-9134-4100-bca3-8b6138e65a89 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:20.840353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-7c388f88-9134-4100-bca3-8b6138e65a89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:20.882964 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c388f88-9134-4100-bca3-8b6138e65a89 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:20.888582 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 862/1727] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:20 2022] POST /identity/v3/projects => generated 371 bytes in 349 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:20.890568 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4686d52-d8fa-4028-8453-11781d7e54fc admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:21.052818 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76a6b88-8475-411d-8778-e5677f0444eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:21.053771 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76a6b88-8475-411d-8778-e5677f0444eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:21.054171 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76a6b88-8475-411d-8778-e5677f0444eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:21.054857 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76a6b88-8475-411d-8778-e5677f0444eb admin admin] PATH_INFO: `/v3/users/543a35f137b747acba1e665e60d6ac4d` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:21.056606 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f76a6b88-8475-411d-8778-e5677f0444eb admin admin] RBAC: Authorizing `identity:delete_user(user_id=543a35f137b747acba1e665e60d6ac4d)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:21.059341 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4686d52-d8fa-4028-8453-11781d7e54fc admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:21.063855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f76a6b88-8475-411d-8778-e5677f0444eb admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:21.066194 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eda3e8e-9aec-4f41-adf7-6246b1413107 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:21.067217 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eda3e8e-9aec-4f41-adf7-6246b1413107 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:21.067657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eda3e8e-9aec-4f41-adf7-6246b1413107 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:21.068083 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eda3e8e-9aec-4f41-adf7-6246b1413107 admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:21.069253 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eda3e8e-9aec-4f41-adf7-6246b1413107 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:20:21.069730 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eda3e8e-9aec-4f41-adf7-6246b1413107 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:21.071942 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eda3e8e-9aec-4f41-adf7-6246b1413107 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:21.103142 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 863/1728] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:20:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:20:21.106020 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eda3e8e-9aec-4f41-adf7-6246b1413107 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:21.142451 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eda3e8e-9aec-4f41-adf7-6246b1413107 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:21.166861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b76152-a128-40ff-bce5-1e988f808f62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:21.168201 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b76152-a128-40ff-bce5-1e988f808f62 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:21.169262 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b76152-a128-40ff-bce5-1e988f808f62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:21.170576 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b76152-a128-40ff-bce5-1e988f808f62 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:21.171565 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b76152-a128-40ff-bce5-1e988f808f62 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:21.177135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b76152-a128-40ff-bce5-1e988f808f62 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:21.215145 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-f76a6b88-8475-411d-8778-e5677f0444eb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '543a35f137b747acba1e665e60d6ac4d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:21.258034 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-f76a6b88-8475-411d-8778-e5677f0444eb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '543a35f137b747acba1e665e60d6ac4d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:21.261763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-f76a6b88-8475-411d-8778-e5677f0444eb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '543a35f137b747acba1e665e60d6ac4d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:21.298590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 864/1729] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:21 2022] POST /identity/v3/users => generated 286 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:21.298845 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b76152-a128-40ff-bce5-1e988f808f62 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:21.307727 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-f76a6b88-8475-411d-8778-e5677f0444eb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '543a35f137b747acba1e665e60d6ac4d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:21.310722 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-98b76152-a128-40ff-bce5-1e988f808f62 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:21.332939 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-f76a6b88-8475-411d-8778-e5677f0444eb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '543a35f137b747acba1e665e60d6ac4d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:21.345073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 866/1730] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:20 2022] DELETE /identity/v3/users/543a35f137b747acba1e665e60d6ac4d => generated 0 bytes in 508 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:21.345073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76a6b88-8475-411d-8778-e5677f0444eb admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:21.345073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-f76a6b88-8475-411d-8778-e5677f0444eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:21.360611 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b76152-a128-40ff-bce5-1e988f808f62 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:21.378470 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:21.475779 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76a6b88-8475-411d-8778-e5677f0444eb admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:21.489125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6470b6c-db0d-4b15-a90a-92b53471e002 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:21.494818 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6470b6c-db0d-4b15-a90a-92b53471e002 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:21.495612 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6470b6c-db0d-4b15-a90a-92b53471e002 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:21.497010 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6470b6c-db0d-4b15-a90a-92b53471e002 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:21.500541 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6470b6c-db0d-4b15-a90a-92b53471e002 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:21.503878 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6470b6c-db0d-4b15-a90a-92b53471e002 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:21.552889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 867/1731] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:21 2022] GET /identity/v3/roles => generated 1420 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:21.556141 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6470b6c-db0d-4b15-a90a-92b53471e002 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:21.612951 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6470b6c-db0d-4b15-a90a-92b53471e002 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:21.631200 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d253e1-50a1-4884-a2e7-e472873e7c50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:21.631869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d253e1-50a1-4884-a2e7-e472873e7c50 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:21.632158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d253e1-50a1-4884-a2e7-e472873e7c50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:21.632434 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d253e1-50a1-4884-a2e7-e472873e7c50 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:21.635247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d253e1-50a1-4884-a2e7-e472873e7c50 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:21.640239 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d253e1-50a1-4884-a2e7-e472873e7c50 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:21.645874 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517c1084-8e43-460c-8a01-0d85c46de722 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:21.649147 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517c1084-8e43-460c-8a01-0d85c46de722 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:21.651808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517c1084-8e43-460c-8a01-0d85c46de722 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:21.652555 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517c1084-8e43-460c-8a01-0d85c46de722 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:21.656319 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517c1084-8e43-460c-8a01-0d85c46de722 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:21.661628 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517c1084-8e43-460c-8a01-0d85c46de722 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:21.682370 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 868/1732] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:21 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:21.685612 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d253e1-50a1-4884-a2e7-e472873e7c50 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:21.723289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d253e1-50a1-4884-a2e7-e472873e7c50 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:21.738485 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9def3fa-bd3f-47d3-a0ae-6010d35d5956 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:21.739176 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9def3fa-bd3f-47d3-a0ae-6010d35d5956 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:21.741969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9def3fa-bd3f-47d3-a0ae-6010d35d5956 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:21.742296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9def3fa-bd3f-47d3-a0ae-6010d35d5956 admin admin] PATH_INFO: `/v3/projects/793630bfbb45489ebfb216dbb8f692a7/users/6264ccfaf98e46a0966381f39eb9769c/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:21.751560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 865/1733] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:21 2022] POST /identity/v3/projects => generated 363 bytes in 455 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:21.755991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517c1084-8e43-460c-8a01-0d85c46de722 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:21.812906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:21.818364 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9def3fa-bd3f-47d3-a0ae-6010d35d5956 admin admin] RBAC: Authorizing `identity:create_grant(project_id=793630bfbb45489ebfb216dbb8f692a7, user_id=6264ccfaf98e46a0966381f39eb9769c, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:21.831118 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9def3fa-bd3f-47d3-a0ae-6010d35d5956 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:21.922185 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 869/1734] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:21 2022] PUT /identity/v3/projects/793630bfbb45489ebfb216dbb8f692a7/users/6264ccfaf98e46a0966381f39eb9769c/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:21.923687 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf34b48b-75df-4047-86ee-6936663ed43d None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:21.924056 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf34b48b-75df-4047-86ee-6936663ed43d None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:21.924336 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf34b48b-75df-4047-86ee-6936663ed43d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:21.998570 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-cf34b48b-75df-4047-86ee-6936663ed43d None None] MFA Rules not processed for user `6264ccfaf98e46a0966381f39eb9769c`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:22.000686 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-cf34b48b-75df-4047-86ee-6936663ed43d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:22.020998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517c1084-8e43-460c-8a01-0d85c46de722 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:22.295217 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715d555e-aa2d-4b31-8a20-2bed5041eb51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:22.295913 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715d555e-aa2d-4b31-8a20-2bed5041eb51 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:22.296190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715d555e-aa2d-4b31-8a20-2bed5041eb51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:22.296463 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715d555e-aa2d-4b31-8a20-2bed5041eb51 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:22.297125 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715d555e-aa2d-4b31-8a20-2bed5041eb51 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:22.299080 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715d555e-aa2d-4b31-8a20-2bed5041eb51 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:22.318666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 870/1735] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:21 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 393 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:22.319678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf34b48b-75df-4047-86ee-6936663ed43d None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:22.347375 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf34b48b-75df-4047-86ee-6936663ed43d None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:22.362484 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8785c384-1148-4e9a-85ad-73663b7fadcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:22.363121 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8785c384-1148-4e9a-85ad-73663b7fadcb admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:22.363607 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8785c384-1148-4e9a-85ad-73663b7fadcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:22.369059 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8785c384-1148-4e9a-85ad-73663b7fadcb admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:22.375383 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8785c384-1148-4e9a-85ad-73663b7fadcb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:22.383137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 866/1736] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:21 2022] POST /identity/v3/users => generated 282 bytes in 628 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:22.383678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8785c384-1148-4e9a-85ad-73663b7fadcb admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:22.388663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715d555e-aa2d-4b31-8a20-2bed5041eb51 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:22.430417 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715d555e-aa2d-4b31-8a20-2bed5041eb51 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:22.448493 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10a90df-f916-4975-bef1-d019e82f2039 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:22.451023 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10a90df-f916-4975-bef1-d019e82f2039 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:22.452938 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10a90df-f916-4975-bef1-d019e82f2039 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:22.454164 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10a90df-f916-4975-bef1-d019e82f2039 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:22.455712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10a90df-f916-4975-bef1-d019e82f2039 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:22.460107 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10a90df-f916-4975-bef1-d019e82f2039 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:22.484231 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 871/1737] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:22 2022] POST /identity/v3/projects => generated 371 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:20:22.488596 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8785c384-1148-4e9a-85ad-73663b7fadcb admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:22.495283 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 867/1738] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:22 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:22.504313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10a90df-f916-4975-bef1-d019e82f2039 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:22.534517 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10a90df-f916-4975-bef1-d019e82f2039 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:22.697685 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:22.728426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8785c384-1148-4e9a-85ad-73663b7fadcb admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:22.758189 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb68ed51-c0d8-4428-bfdc-b48d5b98ae54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:22.759303 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb68ed51-c0d8-4428-bfdc-b48d5b98ae54 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:22.759607 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb68ed51-c0d8-4428-bfdc-b48d5b98ae54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:22.759841 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb68ed51-c0d8-4428-bfdc-b48d5b98ae54 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:22.760492 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb68ed51-c0d8-4428-bfdc-b48d5b98ae54 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:22.762483 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb68ed51-c0d8-4428-bfdc-b48d5b98ae54 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:22.775185 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed880d6a-ae81-4b47-8f48-7fda341c2c0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:22.775829 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed880d6a-ae81-4b47-8f48-7fda341c2c0a admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:22.776100 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed880d6a-ae81-4b47-8f48-7fda341c2c0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:22.783100 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed880d6a-ae81-4b47-8f48-7fda341c2c0a admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:22.783100 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed880d6a-ae81-4b47-8f48-7fda341c2c0a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:22.783100 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed880d6a-ae81-4b47-8f48-7fda341c2c0a admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:22.817344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 868/1739] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:22 2022] GET /identity/v3/roles => generated 1420 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:22.818414 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed880d6a-ae81-4b47-8f48-7fda341c2c0a admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:22.822499 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ed880d6a-ae81-4b47-8f48-7fda341c2c0a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:22.832460 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 872/1740] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:22 2022] POST /identity/v3/users => generated 292 bytes in 341 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:22.832460 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb68ed51-c0d8-4428-bfdc-b48d5b98ae54 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:22.863680 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb68ed51-c0d8-4428-bfdc-b48d5b98ae54 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:22.870351 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed880d6a-ae81-4b47-8f48-7fda341c2c0a admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:22.887675 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6ea3da-cec8-4758-8e5e-3b14c8b10818 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:22.888346 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6ea3da-cec8-4758-8e5e-3b14c8b10818 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:22.888627 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6ea3da-cec8-4758-8e5e-3b14c8b10818 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:22.888901 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6ea3da-cec8-4758-8e5e-3b14c8b10818 admin admin] PATH_INFO: `/v3/projects/5cfa4aef4d854659b5d79e40bb849bc1/users/9d3ef90906ac4db5afbf3c5b9cb55639/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:22.891552 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67e0555-4475-4c2d-836c-72271ce61185 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:22.892457 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67e0555-4475-4c2d-836c-72271ce61185 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:22.892457 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67e0555-4475-4c2d-836c-72271ce61185 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:22.892730 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67e0555-4475-4c2d-836c-72271ce61185 admin admin] PATH_INFO: `/v3/projects/df06e8751c89418db5fb916f63cac9d8` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:22.895297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67e0555-4475-4c2d-836c-72271ce61185 admin admin] RBAC: Authorizing `identity:delete_project(project_id=df06e8751c89418db5fb916f63cac9d8)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:22.897236 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67e0555-4475-4c2d-836c-72271ce61185 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:22.935203 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6ea3da-cec8-4758-8e5e-3b14c8b10818 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5cfa4aef4d854659b5d79e40bb849bc1, user_id=9d3ef90906ac4db5afbf3c5b9cb55639, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:22.938693 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6ea3da-cec8-4758-8e5e-3b14c8b10818 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:22.963443 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 873/1741] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:22 2022] PUT /identity/v3/projects/5cfa4aef4d854659b5d79e40bb849bc1/users/9d3ef90906ac4db5afbf3c5b9cb55639/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:22.964387 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6ea3da-cec8-4758-8e5e-3b14c8b10818 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:22.993713 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=95,keystone:INSERT=3,keystone:DELETE=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:23.015825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-d67e0555-4475-4c2d-836c-72271ce61185 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'df06e8751c89418db5fb916f63cac9d8', 'request_id': 'req-d67e0555-4475-4c2d-836c-72271ce61185'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:23.028038 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-d67e0555-4475-4c2d-836c-72271ce61185 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'df06e8751c89418db5fb916f63cac9d8', 'request_id': 'req-d67e0555-4475-4c2d-836c-72271ce61185'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:23.041987 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-d67e0555-4475-4c2d-836c-72271ce61185 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'df06e8751c89418db5fb916f63cac9d8', 'request_id': 'req-d67e0555-4475-4c2d-836c-72271ce61185'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:23.084295 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-d67e0555-4475-4c2d-836c-72271ce61185 admin admin] The token cache is being invalidate because project df06e8751c89418db5fb916f63cac9d8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:23.085727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-d67e0555-4475-4c2d-836c-72271ce61185 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:23.089807 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-d67e0555-4475-4c2d-836c-72271ce61185 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:23.090427 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 869/1742] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:22 2022] DELETE /identity/v3/projects/df06e8751c89418db5fb916f63cac9d8 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:23.093921 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15622ae1-17aa-40df-b29f-5d1e9072386f None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:23.094524 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15622ae1-17aa-40df-b29f-5d1e9072386f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:23.095028 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15622ae1-17aa-40df-b29f-5d1e9072386f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:23.127011 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6ea3da-cec8-4758-8e5e-3b14c8b10818 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:23.194068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-15622ae1-17aa-40df-b29f-5d1e9072386f None None] MFA Rules not processed for user `9d3ef90906ac4db5afbf3c5b9cb55639`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:23.197822 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-15622ae1-17aa-40df-b29f-5d1e9072386f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:23.256457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2718ca-90a4-4914-bbbd-f3ca6fe7efb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:23.257177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2718ca-90a4-4914-bbbd-f3ca6fe7efb8 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:23.257652 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2718ca-90a4-4914-bbbd-f3ca6fe7efb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:23.258160 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2718ca-90a4-4914-bbbd-f3ca6fe7efb8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:23.258959 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2718ca-90a4-4914-bbbd-f3ca6fe7efb8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:23.260651 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2718ca-90a4-4914-bbbd-f3ca6fe7efb8 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:23.284678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 874/1743] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:22 2022] GET /identity/v3/roles => generated 1420 bytes in 321 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:23.285824 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0b7ca2-bb80-49e5-be02-ccda640e03c0 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:23.289381 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0b7ca2-bb80-49e5-be02-ccda640e03c0 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:23.289712 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0b7ca2-bb80-49e5-be02-ccda640e03c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:23.350470 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-4c0b7ca2-bb80-49e5-be02-ccda640e03c0 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:23.351641 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-4c0b7ca2-bb80-49e5-be02-ccda640e03c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:23.379774 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 870/1744] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:23 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 288 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:23.382263 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15622ae1-17aa-40df-b29f-5d1e9072386f None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:23.388549 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-15622ae1-17aa-40df-b29f-5d1e9072386f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:23.395440 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 875/1745] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:23.397455 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0b7ca2-bb80-49e5-be02-ccda640e03c0 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:23.407530 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-4c0b7ca2-bb80-49e5-be02-ccda640e03c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:23.442199 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15622ae1-17aa-40df-b29f-5d1e9072386f None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:23.445402 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0b7ca2-bb80-49e5-be02-ccda640e03c0 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:23.453828 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91028018-2163-463c-9382-1ead032351d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:23.455511 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca32b0af-8046-410b-a7d7-82f6c298fcb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:23.456298 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca32b0af-8046-410b-a7d7-82f6c298fcb2 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:23.456684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91028018-2163-463c-9382-1ead032351d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:23.457092 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91028018-2163-463c-9382-1ead032351d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:23.457540 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca32b0af-8046-410b-a7d7-82f6c298fcb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:23.458468 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca32b0af-8046-410b-a7d7-82f6c298fcb2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:23.458783 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91028018-2163-463c-9382-1ead032351d1 admin admin] PATH_INFO: `/v3/projects/776f7ceb79e74aa3b1416863ddd4e049/users/3ac1e61411e74cda9076c84304fdab9d/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:23.460440 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca32b0af-8046-410b-a7d7-82f6c298fcb2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:23.465299 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca32b0af-8046-410b-a7d7-82f6c298fcb2 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:23.495688 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91028018-2163-463c-9382-1ead032351d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=776f7ceb79e74aa3b1416863ddd4e049, user_id=3ac1e61411e74cda9076c84304fdab9d, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:23.498403 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91028018-2163-463c-9382-1ead032351d1 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:23.526334 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 871/1746] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:23 2022] PUT /identity/v3/projects/776f7ceb79e74aa3b1416863ddd4e049/users/3ac1e61411e74cda9076c84304fdab9d/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:23.526959 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 876/1747] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:23 2022] POST /identity/v3/projects => generated 363 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:23.528728 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8193819-b425-4c07-af85-2ff698c28028 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:23.529239 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8193819-b425-4c07-af85-2ff698c28028 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:23.529774 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8193819-b425-4c07-af85-2ff698c28028 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:23.546326 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91028018-2163-463c-9382-1ead032351d1 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:23.624909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-a8193819-b425-4c07-af85-2ff698c28028 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:23.627712 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-a8193819-b425-4c07-af85-2ff698c28028 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:23.748258 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:23.788647 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91028018-2163-463c-9382-1ead032351d1 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:23.899006 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 877/1748] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:23.900333 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8193819-b425-4c07-af85-2ff698c28028 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:23.908942 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:DELETE=5,keystone:SELECT=93,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:23.957924 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8193819-b425-4c07-af85-2ff698c28028 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:23.972247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413da963-20ad-462a-8f2d-97049ba466fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:23.975352 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413da963-20ad-462a-8f2d-97049ba466fd admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:23.975645 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413da963-20ad-462a-8f2d-97049ba466fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:23.978035 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413da963-20ad-462a-8f2d-97049ba466fd admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:23.984342 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f48aacd-602e-4f5a-b34c-348aaaace1cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:23.985454 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f48aacd-602e-4f5a-b34c-348aaaace1cd admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:23.985954 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f48aacd-602e-4f5a-b34c-348aaaace1cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:23.986411 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f48aacd-602e-4f5a-b34c-348aaaace1cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:23.987248 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f48aacd-602e-4f5a-b34c-348aaaace1cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:23.989825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f48aacd-602e-4f5a-b34c-348aaaace1cd admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:24.003911 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413da963-20ad-462a-8f2d-97049ba466fd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:24.010807 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413da963-20ad-462a-8f2d-97049ba466fd admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:24.039380 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 872/1749] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:23 2022] GET /identity/v3/roles => generated 1420 bytes in 505 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:24.046806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f48aacd-602e-4f5a-b34c-348aaaace1cd admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:24.089482 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f48aacd-602e-4f5a-b34c-348aaaace1cd admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:24.114410 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c31097-8e09-4088-bbc4-7000b6ead167 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:24.115656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c31097-8e09-4088-bbc4-7000b6ead167 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:24.115656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c31097-8e09-4088-bbc4-7000b6ead167 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:24.115656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c31097-8e09-4088-bbc4-7000b6ead167 admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:24.116204 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c31097-8e09-4088-bbc4-7000b6ead167 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:20:24.118193 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c31097-8e09-4088-bbc4-7000b6ead167 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:24.120289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 878/1750] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:23 2022] POST /identity/v3/users => generated 288 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:24.122461 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413da963-20ad-462a-8f2d-97049ba466fd admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:24.122573 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c31097-8e09-4088-bbc4-7000b6ead167 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:24.143303 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413da963-20ad-462a-8f2d-97049ba466fd admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:24.160970 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6e7caf-5c4b-49a7-893e-6af3b7af1d5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:24.163180 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6e7caf-5c4b-49a7-893e-6af3b7af1d5f admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:24.168676 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6e7caf-5c4b-49a7-893e-6af3b7af1d5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:24.170329 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 873/1751] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:20:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:20:24.170624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6e7caf-5c4b-49a7-893e-6af3b7af1d5f admin admin] PATH_INFO: `/v3/projects/776f7ceb79e74aa3b1416863ddd4e049/users/3ac1e61411e74cda9076c84304fdab9d/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:24.173121 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6e7caf-5c4b-49a7-893e-6af3b7af1d5f admin admin] RBAC: Authorizing `identity:create_grant(project_id=776f7ceb79e74aa3b1416863ddd4e049, user_id=3ac1e61411e74cda9076c84304fdab9d, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:24.177066 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6e7caf-5c4b-49a7-893e-6af3b7af1d5f admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:24.181546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c31097-8e09-4088-bbc4-7000b6ead167 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:24.196144 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 879/1752] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:24 2022] PUT /identity/v3/projects/776f7ceb79e74aa3b1416863ddd4e049/users/3ac1e61411e74cda9076c84304fdab9d/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:24.198415 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6e7caf-5c4b-49a7-893e-6af3b7af1d5f admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:24.215778 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c31097-8e09-4088-bbc4-7000b6ead167 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:24.311869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6e7caf-5c4b-49a7-893e-6af3b7af1d5f admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:24.334733 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f19c31-7704-4a89-950d-2a5172b01151 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:24.337187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f19c31-7704-4a89-950d-2a5172b01151 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:24.337831 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f19c31-7704-4a89-950d-2a5172b01151 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:24.338272 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f19c31-7704-4a89-950d-2a5172b01151 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:24.339490 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f19c31-7704-4a89-950d-2a5172b01151 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:24.342421 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f19c31-7704-4a89-950d-2a5172b01151 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:24.350540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a04b490-2644-4f29-9bdd-3d18c4db12bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:24.351195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a04b490-2644-4f29-9bdd-3d18c4db12bc admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:24.351472 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a04b490-2644-4f29-9bdd-3d18c4db12bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:24.351745 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a04b490-2644-4f29-9bdd-3d18c4db12bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:24.352475 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a04b490-2644-4f29-9bdd-3d18c4db12bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:24.354443 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a04b490-2644-4f29-9bdd-3d18c4db12bc admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:24.377265 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 874/1753] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:24 2022] GET /identity/v3/roles => generated 1420 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:24.378808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07824a43-8a19-4fda-89b4-6bd0784fcbf1 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:24.379096 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07824a43-8a19-4fda-89b4-6bd0784fcbf1 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:24.379366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07824a43-8a19-4fda-89b4-6bd0784fcbf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:24.387989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 880/1754] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:24 2022] POST /identity/v3/projects => generated 365 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:24.388571 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f19c31-7704-4a89-950d-2a5172b01151 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:24.390994 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-a1f19c31-7704-4a89-950d-2a5172b01151 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:24.458533 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-07824a43-8a19-4fda-89b4-6bd0784fcbf1 None None] MFA Rules not processed for user `3ac1e61411e74cda9076c84304fdab9d`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:24.461691 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-07824a43-8a19-4fda-89b4-6bd0784fcbf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:24.592139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f19c31-7704-4a89-950d-2a5172b01151 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:24.609806 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bdfefe-d4f5-4769-9653-be59dbeed87a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:24.610537 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bdfefe-d4f5-4769-9653-be59dbeed87a service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:24.610861 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bdfefe-d4f5-4769-9653-be59dbeed87a service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:24.611150 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bdfefe-d4f5-4769-9653-be59dbeed87a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:24.614522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-f5bdfefe-d4f5-4769-9653-be59dbeed87a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:24.642839 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 875/1755] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:24 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 265 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:24.647179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07824a43-8a19-4fda-89b4-6bd0784fcbf1 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:24.759509 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:24.768739 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:24.829186 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5bdfefe-d4f5-4769-9653-be59dbeed87a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:24.829186 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5bdfefe-d4f5-4769-9653-be59dbeed87a service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:24.835497 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07824a43-8a19-4fda-89b4-6bd0784fcbf1 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:24.848012 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4dee87b-f0a7-4202-9387-981d72d4ade5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:24.849083 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4dee87b-f0a7-4202-9387-981d72d4ade5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:24.849369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4dee87b-f0a7-4202-9387-981d72d4ade5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:24.849763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4dee87b-f0a7-4202-9387-981d72d4ade5 admin admin] PATH_INFO: `/v3/projects/a6722e737b85493db74fd654f295f79d/users/a8ac746745bc47ccbc121a0c378d0e23/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:24.864340 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 881/1756] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:24 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 476 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:24.866067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bdfefe-d4f5-4769-9653-be59dbeed87a service cinder] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:24.887433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4dee87b-f0a7-4202-9387-981d72d4ade5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a6722e737b85493db74fd654f295f79d, user_id=a8ac746745bc47ccbc121a0c378d0e23, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:24.891853 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bdfefe-d4f5-4769-9653-be59dbeed87a service cinder] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:24.892813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4dee87b-f0a7-4202-9387-981d72d4ade5 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:24.910944 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d70bae-fc90-4013-b1ab-5a6e7ab48cce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:24.912569 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d70bae-fc90-4013-b1ab-5a6e7ab48cce admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:24.913156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d70bae-fc90-4013-b1ab-5a6e7ab48cce admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:24.913607 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d70bae-fc90-4013-b1ab-5a6e7ab48cce admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:24.914426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d70bae-fc90-4013-b1ab-5a6e7ab48cce admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:24.916810 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d70bae-fc90-4013-b1ab-5a6e7ab48cce admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:24.939085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 876/1757] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:24 2022] PUT /identity/v3/projects/a6722e737b85493db74fd654f295f79d/users/a8ac746745bc47ccbc121a0c378d0e23/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:24.939085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4dee87b-f0a7-4202-9387-981d72d4ade5 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:24.943900 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-f4dee87b-f0a7-4202-9387-981d72d4ade5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:24.966709 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 882/1758] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:24 2022] POST /identity/v3/users => generated 283 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:24.968221 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d70bae-fc90-4013-b1ab-5a6e7ab48cce admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:24.996490 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d70bae-fc90-4013-b1ab-5a6e7ab48cce admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:25.088468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4dee87b-f0a7-4202-9387-981d72d4ade5 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:25.106767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cbcbb7-e421-4c45-973d-2f2bf217e257 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:25.115680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cbcbb7-e421-4c45-973d-2f2bf217e257 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:25.116131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cbcbb7-e421-4c45-973d-2f2bf217e257 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:25.116937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cbcbb7-e421-4c45-973d-2f2bf217e257 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:25.203218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478db340-dae8-464d-a02d-d58c3b30c040 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:25.203509 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478db340-dae8-464d-a02d-d58c3b30c040 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:25.203593 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478db340-dae8-464d-a02d-d58c3b30c040 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:25.206024 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478db340-dae8-464d-a02d-d58c3b30c040 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:25.211235 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478db340-dae8-464d-a02d-d58c3b30c040 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:25.213165 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478db340-dae8-464d-a02d-d58c3b30c040 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:25.258139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 883/1759] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:24 2022] GET /identity/v3/roles => generated 1420 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:25.260430 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478db340-dae8-464d-a02d-d58c3b30c040 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:25.291780 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478db340-dae8-464d-a02d-d58c3b30c040 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:25.305628 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cbcbb7-e421-4c45-973d-2f2bf217e257 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:25.310191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cbcbb7-e421-4c45-973d-2f2bf217e257 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:25.311348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a535700c-5cab-400d-8726-9a0aa7bb2d69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:25.316105 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a535700c-5cab-400d-8726-9a0aa7bb2d69 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:25.318433 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a535700c-5cab-400d-8726-9a0aa7bb2d69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:25.318433 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a535700c-5cab-400d-8726-9a0aa7bb2d69 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:25.318926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a535700c-5cab-400d-8726-9a0aa7bb2d69 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:25.328302 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a535700c-5cab-400d-8726-9a0aa7bb2d69 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:25.339078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 877/1760] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:24 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:25.360550 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cbcbb7-e421-4c45-973d-2f2bf217e257 service cinder] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:25.365500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 884/1761] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:25 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:25.376021 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a535700c-5cab-400d-8726-9a0aa7bb2d69 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:25.401697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cbcbb7-e421-4c45-973d-2f2bf217e257 service cinder] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:25.421589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a535700c-5cab-400d-8726-9a0aa7bb2d69 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:25.428735 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c07801-4573-446d-97fc-49641d10b9ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:25.430497 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c07801-4573-446d-97fc-49641d10b9ea admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:25.431060 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c07801-4573-446d-97fc-49641d10b9ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:25.431485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c07801-4573-446d-97fc-49641d10b9ea admin admin] PATH_INFO: `/v3/projects/a6722e737b85493db74fd654f295f79d/users/a8ac746745bc47ccbc121a0c378d0e23/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:25.444780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c07801-4573-446d-97fc-49641d10b9ea admin admin] RBAC: Authorizing `identity:create_grant(project_id=a6722e737b85493db74fd654f295f79d, user_id=a8ac746745bc47ccbc121a0c378d0e23, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:25.447612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c07801-4573-446d-97fc-49641d10b9ea admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:25.456176 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d369e12a-5594-4d85-9b58-2c64496c7b01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:25.458318 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d369e12a-5594-4d85-9b58-2c64496c7b01 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:25.458548 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d369e12a-5594-4d85-9b58-2c64496c7b01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:25.458812 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d369e12a-5594-4d85-9b58-2c64496c7b01 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:25.461013 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d369e12a-5594-4d85-9b58-2c64496c7b01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:25.466301 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d369e12a-5594-4d85-9b58-2c64496c7b01 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:25.482591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 878/1762] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:25 2022] PUT /identity/v3/projects/a6722e737b85493db74fd654f295f79d/users/a8ac746745bc47ccbc121a0c378d0e23/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:25.497223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 885/1763] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:25 2022] GET /identity/v3/roles => generated 1420 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:25.501836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c07801-4573-446d-97fc-49641d10b9ea admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:25.510237 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5be887-3458-466f-a3d6-e33240232fcd None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:25.510521 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5be887-3458-466f-a3d6-e33240232fcd None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:25.511461 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5be887-3458-466f-a3d6-e33240232fcd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:25.603046 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-5d5be887-3458-466f-a3d6-e33240232fcd None None] MFA Rules not processed for user `a8ac746745bc47ccbc121a0c378d0e23`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:25.604749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-5d5be887-3458-466f-a3d6-e33240232fcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:25.725790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c07801-4573-446d-97fc-49641d10b9ea admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:25.744030 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86535f06-a326-4931-abca-9139b57faa59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:25.746909 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86535f06-a326-4931-abca-9139b57faa59 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:25.746909 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86535f06-a326-4931-abca-9139b57faa59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:25.747192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86535f06-a326-4931-abca-9139b57faa59 admin admin] PATH_INFO: `/v3/projects/de1f5d9b493942d088cbc5e3925d94c3/users/3fbf83b5c00e46dd83975789618071e8/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:25.759696 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:25.818234 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86535f06-a326-4931-abca-9139b57faa59 admin admin] RBAC: Authorizing `identity:create_grant(project_id=de1f5d9b493942d088cbc5e3925d94c3, user_id=3fbf83b5c00e46dd83975789618071e8, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:25.818234 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86535f06-a326-4931-abca-9139b57faa59 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:25.854438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 886/1764] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:25 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 345 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:25.880280 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 879/1765] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:25 2022] PUT /identity/v3/projects/de1f5d9b493942d088cbc5e3925d94c3/users/3fbf83b5c00e46dd83975789618071e8/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 379 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:25.891106 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818ea160-afdf-42ba-8fd9-83d260da8cd5 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:25.891394 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818ea160-afdf-42ba-8fd9-83d260da8cd5 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:25.892042 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818ea160-afdf-42ba-8fd9-83d260da8cd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:25.934597 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72f622f-3e0e-4d6c-ae0f-0b1b0e5e2ffb None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:25.934932 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72f622f-3e0e-4d6c-ae0f-0b1b0e5e2ffb None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:25.936517 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72f622f-3e0e-4d6c-ae0f-0b1b0e5e2ffb None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:25.944569 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 880/1766] 10.222.0.26 () {54 vars in 921 bytes} [Tue Sep 27 04:20:25 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:20:26.018723 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529a3a9d-021b-4829-bda3-5f01097e45c2 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:26.019013 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529a3a9d-021b-4829-bda3-5f01097e45c2 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:26.019462 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529a3a9d-021b-4829-bda3-5f01097e45c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:26.062789 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-818ea160-afdf-42ba-8fd9-83d260da8cd5 None None] MFA Rules not processed for user `3fbf83b5c00e46dd83975789618071e8`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:26.065830 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-818ea160-afdf-42ba-8fd9-83d260da8cd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:26.103929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-529a3a9d-021b-4829-bda3-5f01097e45c2 None None] MFA Rules not processed for user `a1586ee841cd44658e4d5feb25de7940`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:26.108053 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-529a3a9d-021b-4829-bda3-5f01097e45c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:26.108966 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:26.404531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 887/1767] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:25 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 515 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:26.407216 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818ea160-afdf-42ba-8fd9-83d260da8cd5 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:26.431981 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 881/1768] 10.222.0.26 () {58 vars in 1018 bytes} [Tue Sep 27 04:20:26 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 414 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:26.434151 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529a3a9d-021b-4829-bda3-5f01097e45c2 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:26.591351 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818ea160-afdf-42ba-8fd9-83d260da8cd5 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:26.603921 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c7bb2b-3846-4242-9eb7-230b2cf36ac9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:26.604652 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c7bb2b-3846-4242-9eb7-230b2cf36ac9 service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:26.604936 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c7bb2b-3846-4242-9eb7-230b2cf36ac9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:26.605212 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c7bb2b-3846-4242-9eb7-230b2cf36ac9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:26.622585 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529a3a9d-021b-4829-bda3-5f01097e45c2 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:26.658873 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ae64d4-8589-4c32-a290-3aca0473ffdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:26.660996 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ae64d4-8589-4c32-a290-3aca0473ffdc admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:26.662027 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ae64d4-8589-4c32-a290-3aca0473ffdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:26.662838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ae64d4-8589-4c32-a290-3aca0473ffdc admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:26.664933 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ae64d4-8589-4c32-a290-3aca0473ffdc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:26.673905 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ae64d4-8589-4c32-a290-3aca0473ffdc admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:26.787333 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 882/1769] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:26 2022] POST /identity/v3/projects => generated 363 bytes in 352 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:26.795824 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ae64d4-8589-4c32-a290-3aca0473ffdc admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:26.803961 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c7bb2b-3846-4242-9eb7-230b2cf36ac9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:26.807258 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c7bb2b-3846-4242-9eb7-230b2cf36ac9 service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:26.828129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 888/1770] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:26 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 422 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:26.982597 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ae64d4-8589-4c32-a290-3aca0473ffdc admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:26.998449 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dce209-922d-46d9-abc9-bfe41f1f2d07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:26.999128 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dce209-922d-46d9-abc9-bfe41f1f2d07 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:26.999411 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dce209-922d-46d9-abc9-bfe41f1f2d07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:26.999684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dce209-922d-46d9-abc9-bfe41f1f2d07 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:27.003781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0dce209-922d-46d9-abc9-bfe41f1f2d07 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:27.005784 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0dce209-922d-46d9-abc9-bfe41f1f2d07 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:27.026738 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:27.053920 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 883/1771] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:26 2022] POST /identity/v3/users => generated 288 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:27.055251 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c7bb2b-3846-4242-9eb7-230b2cf36ac9 service cinder] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:27.073958 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c7bb2b-3846-4242-9eb7-230b2cf36ac9 service cinder] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:27.097245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64b6d76-e67e-425e-94a5-26cb12284739 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:27.101435 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64b6d76-e67e-425e-94a5-26cb12284739 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:27.105145 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64b6d76-e67e-425e-94a5-26cb12284739 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:27.105615 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64b6d76-e67e-425e-94a5-26cb12284739 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:27.108122 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64b6d76-e67e-425e-94a5-26cb12284739 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:27.116228 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64b6d76-e67e-425e-94a5-26cb12284739 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:27.120248 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dce209-922d-46d9-abc9-bfe41f1f2d07 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:27.122948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c0dce209-922d-46d9-abc9-bfe41f1f2d07 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:27.165628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 889/1772] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:27 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:27.177247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64b6d76-e67e-425e-94a5-26cb12284739 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:27.208178 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64b6d76-e67e-425e-94a5-26cb12284739 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:27.226912 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a32a5d7-8f24-473a-9991-5b02472b63ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:27.226912 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a32a5d7-8f24-473a-9991-5b02472b63ae admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:27.226912 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a32a5d7-8f24-473a-9991-5b02472b63ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:27.226912 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a32a5d7-8f24-473a-9991-5b02472b63ae admin admin] PATH_INFO: `/v3/projects/a7a58f1304fb4f4493d562f7e2ba707e/users/2502f6864180491faac5ec99e95dba5e/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:27.267429 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a32a5d7-8f24-473a-9991-5b02472b63ae admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7a58f1304fb4f4493d562f7e2ba707e, user_id=2502f6864180491faac5ec99e95dba5e, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:27.270316 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a32a5d7-8f24-473a-9991-5b02472b63ae admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:27.316927 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 890/1773] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:27 2022] PUT /identity/v3/projects/a7a58f1304fb4f4493d562f7e2ba707e/users/2502f6864180491faac5ec99e95dba5e/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:27.318374 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a32a5d7-8f24-473a-9991-5b02472b63ae admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:27.321823 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-8a32a5d7-8f24-473a-9991-5b02472b63ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:27.338969 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dce209-922d-46d9-abc9-bfe41f1f2d07 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:27.408172 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:27.544303 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a32a5d7-8f24-473a-9991-5b02472b63ae admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:27.551767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f8f930-71ab-48cd-9c67-0ea11b00a050 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:27.552712 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f8f930-71ab-48cd-9c67-0ea11b00a050 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:27.553230 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f8f930-71ab-48cd-9c67-0ea11b00a050 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:27.554375 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f8f930-71ab-48cd-9c67-0ea11b00a050 admin admin] PATH_INFO: `/v3/users/0b31e8282eb843e59ad90e37882b8c2d` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:27.554904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd4186b-2f1f-4291-8e7f-0145cb8aed0c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:27.555696 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd4186b-2f1f-4291-8e7f-0145cb8aed0c service nova] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:27.556135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd4186b-2f1f-4291-8e7f-0145cb8aed0c service nova] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:27.557334 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd4186b-2f1f-4291-8e7f-0145cb8aed0c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:27.558103 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f8f930-71ab-48cd-9c67-0ea11b00a050 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0b31e8282eb843e59ad90e37882b8c2d)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:27.561048 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f8f930-71ab-48cd-9c67-0ea11b00a050 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:27.583112 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd4186b-2f1f-4291-8e7f-0145cb8aed0c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:27.585069 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd4186b-2f1f-4291-8e7f-0145cb8aed0c service nova] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:27.608001 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 884/1774] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:27 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 487 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:20:27.608001 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd4186b-2f1f-4291-8e7f-0145cb8aed0c service nova] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:27.627724 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd4186b-2f1f-4291-8e7f-0145cb8aed0c service nova] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:27.645313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8370c0-8108-4ec8-814f-524af13d69ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:27.647099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8370c0-8108-4ec8-814f-524af13d69ab admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:27.647099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8370c0-8108-4ec8-814f-524af13d69ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:27.648130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8370c0-8108-4ec8-814f-524af13d69ab admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:27.649076 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8370c0-8108-4ec8-814f-524af13d69ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:27.661317 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8370c0-8108-4ec8-814f-524af13d69ab admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:27.690148 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 885/1775] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:27 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:27.690939 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8370c0-8108-4ec8-814f-524af13d69ab admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:27.707745 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8370c0-8108-4ec8-814f-524af13d69ab admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:27.721927 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebd7869-e18a-4851-ac85-e3b924a38937 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:27.722580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebd7869-e18a-4851-ac85-e3b924a38937 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:27.722864 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebd7869-e18a-4851-ac85-e3b924a38937 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:27.723145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebd7869-e18a-4851-ac85-e3b924a38937 admin admin] PATH_INFO: `/v3/users/6264ccfaf98e46a0966381f39eb9769c` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:27.724410 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebd7869-e18a-4851-ac85-e3b924a38937 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6264ccfaf98e46a0966381f39eb9769c)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:27.725952 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-77f8f930-71ab-48cd-9c67-0ea11b00a050 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0b31e8282eb843e59ad90e37882b8c2d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:27.726578 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebd7869-e18a-4851-ac85-e3b924a38937 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:27.760050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-77f8f930-71ab-48cd-9c67-0ea11b00a050 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0b31e8282eb843e59ad90e37882b8c2d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:27.764162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-77f8f930-71ab-48cd-9c67-0ea11b00a050 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0b31e8282eb843e59ad90e37882b8c2d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:27.785628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-77f8f930-71ab-48cd-9c67-0ea11b00a050 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0b31e8282eb843e59ad90e37882b8c2d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:27.811308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-77f8f930-71ab-48cd-9c67-0ea11b00a050 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0b31e8282eb843e59ad90e37882b8c2d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:27.817228 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-3ebd7869-e18a-4851-ac85-e3b924a38937 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6264ccfaf98e46a0966381f39eb9769c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:27.820652 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 891/1776] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:27 2022] DELETE /identity/v3/users/0b31e8282eb843e59ad90e37882b8c2d => generated 0 bytes in 503 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:27.822129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f8f930-71ab-48cd-9c67-0ea11b00a050 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:27.823835 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-77f8f930-71ab-48cd-9c67-0ea11b00a050 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:27.830614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-3ebd7869-e18a-4851-ac85-e3b924a38937 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6264ccfaf98e46a0966381f39eb9769c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:27.833044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-3ebd7869-e18a-4851-ac85-e3b924a38937 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6264ccfaf98e46a0966381f39eb9769c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:27.835105 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:INSERT=3,keystone:SELECT=90,keystone:DELETE=7 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:27.857913 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-3ebd7869-e18a-4851-ac85-e3b924a38937 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6264ccfaf98e46a0966381f39eb9769c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:27.885956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-3ebd7869-e18a-4851-ac85-e3b924a38937 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6264ccfaf98e46a0966381f39eb9769c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:27.902051 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 886/1777] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:27 2022] DELETE /identity/v3/users/6264ccfaf98e46a0966381f39eb9769c => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:27.904323 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdccd7f-14e7-403f-b53d-4cf99a41498f None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:27.907052 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdccd7f-14e7-403f-b53d-4cf99a41498f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:27.907570 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdccd7f-14e7-403f-b53d-4cf99a41498f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:27.964401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-7fdccd7f-14e7-403f-b53d-4cf99a41498f None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:27.969081 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-7fdccd7f-14e7-403f-b53d-4cf99a41498f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:27.970747 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f8f930-71ab-48cd-9c67-0ea11b00a050 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:28.072970 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:28.085825 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457b155d-e153-43dc-a60e-0ffb4ac875c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:28.086826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457b155d-e153-43dc-a60e-0ffb4ac875c3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:28.088679 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457b155d-e153-43dc-a60e-0ffb4ac875c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:28.111748 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457b155d-e153-43dc-a60e-0ffb4ac875c3 admin admin] PATH_INFO: `/v3/projects/a7a58f1304fb4f4493d562f7e2ba707e/users/2502f6864180491faac5ec99e95dba5e/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:28.144391 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-457b155d-e153-43dc-a60e-0ffb4ac875c3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7a58f1304fb4f4493d562f7e2ba707e, user_id=2502f6864180491faac5ec99e95dba5e, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:28.157440 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-457b155d-e153-43dc-a60e-0ffb4ac875c3 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:28.160288 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 887/1778] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 257 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:28.172469 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c83038-4b20-4206-81ff-383a360df105 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:28.172812 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c83038-4b20-4206-81ff-383a360df105 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:28.178016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c83038-4b20-4206-81ff-383a360df105 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:28.227929 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 892/1779] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:27 2022] PUT /identity/v3/projects/a7a58f1304fb4f4493d562f7e2ba707e/users/2502f6864180491faac5ec99e95dba5e/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 395 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:28.228965 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457b155d-e153-43dc-a60e-0ffb4ac875c3 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:28.243267 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-457b155d-e153-43dc-a60e-0ffb4ac875c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:28.266530 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-a6c83038-4b20-4206-81ff-383a360df105 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:28.267739 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a6c83038-4b20-4206-81ff-383a360df105 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:28.426356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457b155d-e153-43dc-a60e-0ffb4ac875c3 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:28.436872 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90ec595-4f1a-4a1e-9371-f85a29f424f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:28.444688 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 888/1780] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 284 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:28.451093 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c83038-4b20-4206-81ff-383a360df105 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:28.451093 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a6c83038-4b20-4206-81ff-383a360df105 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:28.461275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90ec595-4f1a-4a1e-9371-f85a29f424f9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:28.472842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90ec595-4f1a-4a1e-9371-f85a29f424f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:28.473413 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90ec595-4f1a-4a1e-9371-f85a29f424f9 admin admin] PATH_INFO: `/v3/users/9d3ef90906ac4db5afbf3c5b9cb55639` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:28.481207 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90ec595-4f1a-4a1e-9371-f85a29f424f9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9d3ef90906ac4db5afbf3c5b9cb55639)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:28.487022 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90ec595-4f1a-4a1e-9371-f85a29f424f9 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:28.648045 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-b90ec595-4f1a-4a1e-9371-f85a29f424f9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9d3ef90906ac4db5afbf3c5b9cb55639'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:28.665020 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c83038-4b20-4206-81ff-383a360df105 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:28.670414 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-b90ec595-4f1a-4a1e-9371-f85a29f424f9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9d3ef90906ac4db5afbf3c5b9cb55639'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:28.671964 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-b90ec595-4f1a-4a1e-9371-f85a29f424f9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9d3ef90906ac4db5afbf3c5b9cb55639'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:28.674556 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea1653e-c22f-4cf3-9654-0556ad523561 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:28.675304 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea1653e-c22f-4cf3-9654-0556ad523561 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:28.675694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea1653e-c22f-4cf3-9654-0556ad523561 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:28.676086 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea1653e-c22f-4cf3-9654-0556ad523561 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:28.680981 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-2ea1653e-c22f-4cf3-9654-0556ad523561 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:28.689576 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-b90ec595-4f1a-4a1e-9371-f85a29f424f9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9d3ef90906ac4db5afbf3c5b9cb55639'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:28.711503 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea1653e-c22f-4cf3-9654-0556ad523561 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:28.719200 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea1653e-c22f-4cf3-9654-0556ad523561 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:28.719393 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-b90ec595-4f1a-4a1e-9371-f85a29f424f9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9d3ef90906ac4db5afbf3c5b9cb55639'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:28.736962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 893/1781] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:28 2022] DELETE /identity/v3/users/9d3ef90906ac4db5afbf3c5b9cb55639 => generated 0 bytes in 508 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:28.741499 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ee1328-39b7-4220-b4de-0489703aa020 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:28.741877 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ee1328-39b7-4220-b4de-0489703aa020 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:28.742153 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ee1328-39b7-4220-b4de-0489703aa020 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:28.751579 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 889/1782] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:28.752988 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea1653e-c22f-4cf3-9654-0556ad523561 service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:28.754921 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-2ea1653e-c22f-4cf3-9654-0556ad523561 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:28.775366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea1653e-c22f-4cf3-9654-0556ad523561 service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:28.805521 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:INSERT=1,keystone:DELETE=3,keystone:SELECT=96 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:28.894973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-50ee1328-39b7-4220-b4de-0489703aa020 None None] MFA Rules not processed for user `2502f6864180491faac5ec99e95dba5e`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:28.902758 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-50ee1328-39b7-4220-b4de-0489703aa020 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:28.904384 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c631d0-6396-431c-a7ff-304d441cd030 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:28.906739 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c631d0-6396-431c-a7ff-304d441cd030 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:28.908339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c631d0-6396-431c-a7ff-304d441cd030 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:28.909251 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c631d0-6396-431c-a7ff-304d441cd030 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:28.911620 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-67c631d0-6396-431c-a7ff-304d441cd030 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:29.072826 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c631d0-6396-431c-a7ff-304d441cd030 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:29.075221 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c631d0-6396-431c-a7ff-304d441cd030 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:29.103312 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 890/1783] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:29.105290 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81c6b62-8d6b-4305-a168-8fc1376f7e63 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:29.105645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81c6b62-8d6b-4305-a168-8fc1376f7e63 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:29.106797 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81c6b62-8d6b-4305-a168-8fc1376f7e63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:29.116255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:29.156359 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-b81c6b62-8d6b-4305-a168-8fc1376f7e63 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:29.160114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-b81c6b62-8d6b-4305-a168-8fc1376f7e63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:29.194106 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 891/1784] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:29.206395 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 894/1785] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:28 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 469 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:29.219801 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81c6b62-8d6b-4305-a168-8fc1376f7e63 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:29.221590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-b81c6b62-8d6b-4305-a168-8fc1376f7e63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:29.320799 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ee1328-39b7-4220-b4de-0489703aa020 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:29.335888 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ee1328-39b7-4220-b4de-0489703aa020 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:29.345915 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be72caeb-4ef7-41e9-9675-8d3f196e897b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:29.346612 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be72caeb-4ef7-41e9-9675-8d3f196e897b service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:29.348077 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be72caeb-4ef7-41e9-9675-8d3f196e897b service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:29.348374 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be72caeb-4ef7-41e9-9675-8d3f196e897b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:29.363267 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be72caeb-4ef7-41e9-9675-8d3f196e897b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:29.365596 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be72caeb-4ef7-41e9-9675-8d3f196e897b service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:29.377277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81c6b62-8d6b-4305-a168-8fc1376f7e63 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:29.377993 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 895/1786] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:20:29.378863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be72caeb-4ef7-41e9-9675-8d3f196e897b service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:29.380659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-be72caeb-4ef7-41e9-9675-8d3f196e897b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:29.387634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1980de04-8294-4097-8a83-0cfcb0e3f4ba service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:29.388267 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1980de04-8294-4097-8a83-0cfcb0e3f4ba service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:29.388551 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1980de04-8294-4097-8a83-0cfcb0e3f4ba service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:29.391929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1980de04-8294-4097-8a83-0cfcb0e3f4ba service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:29.410510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be72caeb-4ef7-41e9-9675-8d3f196e897b service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:29.427679 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1980de04-8294-4097-8a83-0cfcb0e3f4ba service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:29.429164 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1980de04-8294-4097-8a83-0cfcb0e3f4ba service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:29.431081 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8c9763-46ee-432a-8c31-cb19db3d0511 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:29.435203 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8c9763-46ee-432a-8c31-cb19db3d0511 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:29.437183 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8c9763-46ee-432a-8c31-cb19db3d0511 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:29.437725 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8c9763-46ee-432a-8c31-cb19db3d0511 admin admin] PATH_INFO: `/v3/users/2f5d29f9f1814716a43ec5dcfeb5c747` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:29.442398 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8c9763-46ee-432a-8c31-cb19db3d0511 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2f5d29f9f1814716a43ec5dcfeb5c747)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:29.447782 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8c9763-46ee-432a-8c31-cb19db3d0511 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:29.452685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 892/1787] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:29 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 233 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:29.552245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-1b8c9763-46ee-432a-8c31-cb19db3d0511 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2f5d29f9f1814716a43ec5dcfeb5c747'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:29.585340 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-1b8c9763-46ee-432a-8c31-cb19db3d0511 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2f5d29f9f1814716a43ec5dcfeb5c747'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:29.586040 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-1b8c9763-46ee-432a-8c31-cb19db3d0511 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2f5d29f9f1814716a43ec5dcfeb5c747'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:29.600376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-1b8c9763-46ee-432a-8c31-cb19db3d0511 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2f5d29f9f1814716a43ec5dcfeb5c747'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:29.603687 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1980de04-8294-4097-8a83-0cfcb0e3f4ba service cinder] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:29.615043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1980de04-8294-4097-8a83-0cfcb0e3f4ba service cinder] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:29.619926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5532e73-f2a3-49b6-80d2-8502cca1c6a2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:29.620849 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5532e73-f2a3-49b6-80d2-8502cca1c6a2 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:29.621329 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5532e73-f2a3-49b6-80d2-8502cca1c6a2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:29.621860 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5532e73-f2a3-49b6-80d2-8502cca1c6a2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:29.626708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c5532e73-f2a3-49b6-80d2-8502cca1c6a2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:29.645334 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-1b8c9763-46ee-432a-8c31-cb19db3d0511 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2f5d29f9f1814716a43ec5dcfeb5c747'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:29.663425 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 896/1788] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:29 2022] DELETE /identity/v3/users/2f5d29f9f1814716a43ec5dcfeb5c747 => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:29.670110 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3794bdc6-d5b2-409e-8790-79bdc9972781 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:29.671258 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3794bdc6-d5b2-409e-8790-79bdc9972781 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:29.671780 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3794bdc6-d5b2-409e-8790-79bdc9972781 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:29.703876 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-3794bdc6-d5b2-409e-8790-79bdc9972781 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:29.706411 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-3794bdc6-d5b2-409e-8790-79bdc9972781 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:29.858795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5532e73-f2a3-49b6-80d2-8502cca1c6a2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:29.860240 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5532e73-f2a3-49b6-80d2-8502cca1c6a2 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:29.871431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 893/1789] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:29 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 268 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:20:29.872469 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 897/1790] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:29.901036 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5532e73-f2a3-49b6-80d2-8502cca1c6a2 service cinder] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:29.903002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c5532e73-f2a3-49b6-80d2-8502cca1c6a2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:29.912687 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:29.928084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5532e73-f2a3-49b6-80d2-8502cca1c6a2 service cinder] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:30.056570 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07919e21-c860-4300-80d5-84117f16d1cb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:30.057959 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07919e21-c860-4300-80d5-84117f16d1cb service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:30.060755 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07919e21-c860-4300-80d5-84117f16d1cb service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:30.061125 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07919e21-c860-4300-80d5-84117f16d1cb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:30.088601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07919e21-c860-4300-80d5-84117f16d1cb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:30.094378 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07919e21-c860-4300-80d5-84117f16d1cb service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:30.120971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 894/1791] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:30.525398 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3794bdc6-d5b2-409e-8790-79bdc9972781 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:30.529148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-3794bdc6-d5b2-409e-8790-79bdc9972781 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:30.548362 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3794bdc6-d5b2-409e-8790-79bdc9972781 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:30.555462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0303f58f-1582-4323-becb-f1fa2daa041b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:30.556229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0303f58f-1582-4323-becb-f1fa2daa041b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:30.556584 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0303f58f-1582-4323-becb-f1fa2daa041b admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:30.556931 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0303f58f-1582-4323-becb-f1fa2daa041b admin admin] PATH_INFO: `/v3/projects/52f37847ffde4c47a8716bff540866c1` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:30.559682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0303f58f-1582-4323-becb-f1fa2daa041b admin admin] RBAC: Authorizing `identity:delete_project(project_id=52f37847ffde4c47a8716bff540866c1)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:30.564109 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0303f58f-1582-4323-becb-f1fa2daa041b admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:30.657702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:30.660037 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0303f58f-1582-4323-becb-f1fa2daa041b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '52f37847ffde4c47a8716bff540866c1', 'request_id': 'req-0303f58f-1582-4323-becb-f1fa2daa041b'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:30.670814 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0303f58f-1582-4323-becb-f1fa2daa041b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '52f37847ffde4c47a8716bff540866c1', 'request_id': 'req-0303f58f-1582-4323-becb-f1fa2daa041b'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:30.673390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07919e21-c860-4300-80d5-84117f16d1cb service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:30.675157 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-07919e21-c860-4300-80d5-84117f16d1cb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:30.697976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0303f58f-1582-4323-becb-f1fa2daa041b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '52f37847ffde4c47a8716bff540866c1', 'request_id': 'req-0303f58f-1582-4323-becb-f1fa2daa041b'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:30.698834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07919e21-c860-4300-80d5-84117f16d1cb service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:30.719757 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0303f58f-1582-4323-becb-f1fa2daa041b admin admin] The token cache is being invalidate because project 52f37847ffde4c47a8716bff540866c1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:30.720156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0303f58f-1582-4323-becb-f1fa2daa041b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:30.721874 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0303f58f-1582-4323-becb-f1fa2daa041b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:30.724458 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 898/1792] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:30 2022] DELETE /identity/v3/projects/52f37847ffde4c47a8716bff540866c1 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:30.725493 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0303f58f-1582-4323-becb-f1fa2daa041b admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:30.735014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0303f58f-1582-4323-becb-f1fa2daa041b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:30.790172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c2c16c-1b3a-472a-940f-e4487c2f31e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:30.790818 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c2c16c-1b3a-472a-940f-e4487c2f31e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:30.791096 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c2c16c-1b3a-472a-940f-e4487c2f31e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:30.791376 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c2c16c-1b3a-472a-940f-e4487c2f31e3 admin admin] PATH_INFO: `/v3/projects/793630bfbb45489ebfb216dbb8f692a7` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:30.792727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c2c16c-1b3a-472a-940f-e4487c2f31e3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=793630bfbb45489ebfb216dbb8f692a7)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:30.794660 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c2c16c-1b3a-472a-940f-e4487c2f31e3 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:30.844211 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0303f58f-1582-4323-becb-f1fa2daa041b admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:30.851293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f447f9-64a3-43a3-b60d-a72406daafd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:30.852509 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f447f9-64a3-43a3-b60d-a72406daafd4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:30.852929 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f447f9-64a3-43a3-b60d-a72406daafd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:30.853243 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f447f9-64a3-43a3-b60d-a72406daafd4 admin admin] PATH_INFO: `/v3/projects/5cfa4aef4d854659b5d79e40bb849bc1` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:30.854446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f447f9-64a3-43a3-b60d-a72406daafd4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5cfa4aef4d854659b5d79e40bb849bc1)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:30.857339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f447f9-64a3-43a3-b60d-a72406daafd4 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:30.941688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-36f447f9-64a3-43a3-b60d-a72406daafd4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5cfa4aef4d854659b5d79e40bb849bc1', 'request_id': 'req-36f447f9-64a3-43a3-b60d-a72406daafd4'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:30.943129 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a4c2c16c-1b3a-472a-940f-e4487c2f31e3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '793630bfbb45489ebfb216dbb8f692a7', 'request_id': 'req-a4c2c16c-1b3a-472a-940f-e4487c2f31e3'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:30.950074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-36f447f9-64a3-43a3-b60d-a72406daafd4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5cfa4aef4d854659b5d79e40bb849bc1', 'request_id': 'req-36f447f9-64a3-43a3-b60d-a72406daafd4'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:30.951833 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a4c2c16c-1b3a-472a-940f-e4487c2f31e3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '793630bfbb45489ebfb216dbb8f692a7', 'request_id': 'req-a4c2c16c-1b3a-472a-940f-e4487c2f31e3'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:30.958931 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-36f447f9-64a3-43a3-b60d-a72406daafd4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5cfa4aef4d854659b5d79e40bb849bc1', 'request_id': 'req-36f447f9-64a3-43a3-b60d-a72406daafd4'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:30.961657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a4c2c16c-1b3a-472a-940f-e4487c2f31e3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '793630bfbb45489ebfb216dbb8f692a7', 'request_id': 'req-a4c2c16c-1b3a-472a-940f-e4487c2f31e3'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:30.981708 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-36f447f9-64a3-43a3-b60d-a72406daafd4 admin admin] The token cache is being invalidate because project 5cfa4aef4d854659b5d79e40bb849bc1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:30.982107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-36f447f9-64a3-43a3-b60d-a72406daafd4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:30.983073 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a4c2c16c-1b3a-472a-940f-e4487c2f31e3 admin admin] The token cache is being invalidate because project 793630bfbb45489ebfb216dbb8f692a7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:30.984081 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a4c2c16c-1b3a-472a-940f-e4487c2f31e3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:30.984795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a4c2c16c-1b3a-472a-940f-e4487c2f31e3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:30.986241 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-36f447f9-64a3-43a3-b60d-a72406daafd4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:30.987161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 895/1793] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:30 2022] DELETE /identity/v3/projects/793630bfbb45489ebfb216dbb8f692a7 => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:30.988135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c2c16c-1b3a-472a-940f-e4487c2f31e3 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:30.988688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 899/1794] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:30 2022] DELETE /identity/v3/projects/5cfa4aef4d854659b5d79e40bb849bc1 => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:30.990263 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f447f9-64a3-43a3-b60d-a72406daafd4 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:30.992215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-36f447f9-64a3-43a3-b60d-a72406daafd4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:30.992769 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a4c2c16c-1b3a-472a-940f-e4487c2f31e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:31.057648 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:31.084209 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f447f9-64a3-43a3-b60d-a72406daafd4 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:31.090166 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d97fb57-7217-4427-826c-83992751c66e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:31.090989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d97fb57-7217-4427-826c-83992751c66e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:31.091394 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d97fb57-7217-4427-826c-83992751c66e admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:31.091768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d97fb57-7217-4427-826c-83992751c66e admin admin] PATH_INFO: `/v3/projects/1abc25ba289d49f285f776cd65e3e3ea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:31.094313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d97fb57-7217-4427-826c-83992751c66e admin admin] RBAC: Authorizing `identity:delete_project(project_id=1abc25ba289d49f285f776cd65e3e3ea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:31.096621 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d97fb57-7217-4427-826c-83992751c66e admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:31.108029 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:DELETE=11,keystone:SELECT=87,keystone:INSERT=2 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:31.120470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c2c16c-1b3a-472a-940f-e4487c2f31e3 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:31.125980 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ea5202-953b-482a-bdba-c043b17ec5d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:31.126929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ea5202-953b-482a-bdba-c043b17ec5d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:31.127420 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ea5202-953b-482a-bdba-c043b17ec5d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:31.127864 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ea5202-953b-482a-bdba-c043b17ec5d9 admin admin] PATH_INFO: `/v3/users/6a15b1818fb442138bf9d479639978ce` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:31.129542 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ea5202-953b-482a-bdba-c043b17ec5d9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6a15b1818fb442138bf9d479639978ce)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:31.131827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ea5202-953b-482a-bdba-c043b17ec5d9 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:31.175891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0d97fb57-7217-4427-826c-83992751c66e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1abc25ba289d49f285f776cd65e3e3ea', 'request_id': 'req-0d97fb57-7217-4427-826c-83992751c66e'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.181597 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0d97fb57-7217-4427-826c-83992751c66e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1abc25ba289d49f285f776cd65e3e3ea', 'request_id': 'req-0d97fb57-7217-4427-826c-83992751c66e'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.186069 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a9ea5202-953b-482a-bdba-c043b17ec5d9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6a15b1818fb442138bf9d479639978ce'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.190473 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0d97fb57-7217-4427-826c-83992751c66e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1abc25ba289d49f285f776cd65e3e3ea', 'request_id': 'req-0d97fb57-7217-4427-826c-83992751c66e'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.194421 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a9ea5202-953b-482a-bdba-c043b17ec5d9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6a15b1818fb442138bf9d479639978ce'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.195504 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a9ea5202-953b-482a-bdba-c043b17ec5d9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6a15b1818fb442138bf9d479639978ce'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.203409 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a9ea5202-953b-482a-bdba-c043b17ec5d9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6a15b1818fb442138bf9d479639978ce'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.205423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0d97fb57-7217-4427-826c-83992751c66e admin admin] The token cache is being invalidate because project 1abc25ba289d49f285f776cd65e3e3ea was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:31.205718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0d97fb57-7217-4427-826c-83992751c66e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.206455 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0d97fb57-7217-4427-826c-83992751c66e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.208176 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 900/1795] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:30 2022] DELETE /identity/v3/projects/1abc25ba289d49f285f776cd65e3e3ea => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:31.208729 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d97fb57-7217-4427-826c-83992751c66e admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:31.210548 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0d97fb57-7217-4427-826c-83992751c66e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:31.217493 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a9ea5202-953b-482a-bdba-c043b17ec5d9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6a15b1818fb442138bf9d479639978ce'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.226585 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 896/1796] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:30 2022] DELETE /identity/v3/users/6a15b1818fb442138bf9d479639978ce => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:31.226585 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ea5202-953b-482a-bdba-c043b17ec5d9 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:31.227946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a9ea5202-953b-482a-bdba-c043b17ec5d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:31.300335 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d97fb57-7217-4427-826c-83992751c66e admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:31.319393 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ea5202-953b-482a-bdba-c043b17ec5d9 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:31.325599 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba652ba5-323c-4813-ab62-5da6365a0491 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:31.326812 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba652ba5-323c-4813-ab62-5da6365a0491 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:31.327184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba652ba5-323c-4813-ab62-5da6365a0491 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:31.327538 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba652ba5-323c-4813-ab62-5da6365a0491 admin admin] PATH_INFO: `/v3/users/a8ac746745bc47ccbc121a0c378d0e23` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:31.329067 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba652ba5-323c-4813-ab62-5da6365a0491 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a8ac746745bc47ccbc121a0c378d0e23)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:31.331293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba652ba5-323c-4813-ab62-5da6365a0491 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:31.420512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-ba652ba5-323c-4813-ab62-5da6365a0491 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a8ac746745bc47ccbc121a0c378d0e23'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.433585 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-ba652ba5-323c-4813-ab62-5da6365a0491 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a8ac746745bc47ccbc121a0c378d0e23'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.435901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-ba652ba5-323c-4813-ab62-5da6365a0491 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a8ac746745bc47ccbc121a0c378d0e23'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.439337 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:31.445483 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a6394b-5c44-4136-8383-68df0880a11a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:31.449382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a6394b-5c44-4136-8383-68df0880a11a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:31.449712 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a6394b-5c44-4136-8383-68df0880a11a admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:31.450129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a6394b-5c44-4136-8383-68df0880a11a admin admin] PATH_INFO: `/v3/users/3ac1e61411e74cda9076c84304fdab9d` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:31.450244 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-ba652ba5-323c-4813-ab62-5da6365a0491 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a8ac746745bc47ccbc121a0c378d0e23'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.451588 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a6394b-5c44-4136-8383-68df0880a11a admin admin] RBAC: Authorizing `identity:delete_user(user_id=3ac1e61411e74cda9076c84304fdab9d)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:31.456454 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a6394b-5c44-4136-8383-68df0880a11a admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:31.469374 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-ba652ba5-323c-4813-ab62-5da6365a0491 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a8ac746745bc47ccbc121a0c378d0e23'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.477347 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 897/1797] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:31 2022] DELETE /identity/v3/users/a8ac746745bc47ccbc121a0c378d0e23 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:31.478289 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba652ba5-323c-4813-ab62-5da6365a0491 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:31.479637 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ba652ba5-323c-4813-ab62-5da6365a0491 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:31.571775 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:31.581589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-39a6394b-5c44-4136-8383-68df0880a11a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3ac1e61411e74cda9076c84304fdab9d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.597530 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-39a6394b-5c44-4136-8383-68df0880a11a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3ac1e61411e74cda9076c84304fdab9d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.598412 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-39a6394b-5c44-4136-8383-68df0880a11a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3ac1e61411e74cda9076c84304fdab9d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.611561 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-39a6394b-5c44-4136-8383-68df0880a11a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3ac1e61411e74cda9076c84304fdab9d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.626135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba652ba5-323c-4813-ab62-5da6365a0491 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:31.629060 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-39a6394b-5c44-4136-8383-68df0880a11a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3ac1e61411e74cda9076c84304fdab9d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.637295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 901/1798] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:31 2022] DELETE /identity/v3/users/3ac1e61411e74cda9076c84304fdab9d => generated 0 bytes in 429 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:31.637922 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a6394b-5c44-4136-8383-68df0880a11a admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:31.640046 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-39a6394b-5c44-4136-8383-68df0880a11a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:31.642807 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029d1661-ebcc-4031-a993-d7d74ab2a0cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:31.649867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029d1661-ebcc-4031-a993-d7d74ab2a0cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:31.649867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029d1661-ebcc-4031-a993-d7d74ab2a0cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:31.649867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029d1661-ebcc-4031-a993-d7d74ab2a0cf admin admin] PATH_INFO: `/v3/users/5827d0b3c9f64f5cacef8b76c3c550ec` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:31.649867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029d1661-ebcc-4031-a993-d7d74ab2a0cf admin admin] RBAC: Authorizing `identity:delete_user(user_id=5827d0b3c9f64f5cacef8b76c3c550ec)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:31.649867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029d1661-ebcc-4031-a993-d7d74ab2a0cf admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:31.741197 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-029d1661-ebcc-4031-a993-d7d74ab2a0cf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5827d0b3c9f64f5cacef8b76c3c550ec'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.751984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-029d1661-ebcc-4031-a993-d7d74ab2a0cf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5827d0b3c9f64f5cacef8b76c3c550ec'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.761414 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-029d1661-ebcc-4031-a993-d7d74ab2a0cf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5827d0b3c9f64f5cacef8b76c3c550ec'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.775443 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-029d1661-ebcc-4031-a993-d7d74ab2a0cf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5827d0b3c9f64f5cacef8b76c3c550ec'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.803133 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a6394b-5c44-4136-8383-68df0880a11a admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:31.810091 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-029d1661-ebcc-4031-a993-d7d74ab2a0cf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5827d0b3c9f64f5cacef8b76c3c550ec'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.814002 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879c1089-e463-4506-9a0b-7be5d0436e00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:31.815187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879c1089-e463-4506-9a0b-7be5d0436e00 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:31.815698 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879c1089-e463-4506-9a0b-7be5d0436e00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:31.816166 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879c1089-e463-4506-9a0b-7be5d0436e00 admin admin] PATH_INFO: `/v3/projects/d7f601e71ace4199be72b41bba1bbe83` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:31.817960 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 898/1799] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:31 2022] DELETE /identity/v3/users/5827d0b3c9f64f5cacef8b76c3c550ec => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:31.818776 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-879c1089-e463-4506-9a0b-7be5d0436e00 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d7f601e71ace4199be72b41bba1bbe83)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:31.820962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-879c1089-e463-4506-9a0b-7be5d0436e00 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:31.923224 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-879c1089-e463-4506-9a0b-7be5d0436e00 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd7f601e71ace4199be72b41bba1bbe83', 'request_id': 'req-879c1089-e463-4506-9a0b-7be5d0436e00'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.929290 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-879c1089-e463-4506-9a0b-7be5d0436e00 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd7f601e71ace4199be72b41bba1bbe83', 'request_id': 'req-879c1089-e463-4506-9a0b-7be5d0436e00'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.939073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-879c1089-e463-4506-9a0b-7be5d0436e00 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd7f601e71ace4199be72b41bba1bbe83', 'request_id': 'req-879c1089-e463-4506-9a0b-7be5d0436e00'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.984804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-879c1089-e463-4506-9a0b-7be5d0436e00 admin admin] The token cache is being invalidate because project d7f601e71ace4199be72b41bba1bbe83 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:31.985568 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-879c1089-e463-4506-9a0b-7be5d0436e00 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.987849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-879c1089-e463-4506-9a0b-7be5d0436e00 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:31.991290 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 902/1800] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:31 2022] DELETE /identity/v3/projects/d7f601e71ace4199be72b41bba1bbe83 => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:31.999114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb9f5d0-7771-4cef-a9cf-b60b806e7875 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:31.999604 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb9f5d0-7771-4cef-a9cf-b60b806e7875 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:31.999678 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb9f5d0-7771-4cef-a9cf-b60b806e7875 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:32.047332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879c1089-e463-4506-9a0b-7be5d0436e00 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:32.047332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-879c1089-e463-4506-9a0b-7be5d0436e00 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:32.053180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-ecb9f5d0-7771-4cef-a9cf-b60b806e7875 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:32.053180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ecb9f5d0-7771-4cef-a9cf-b60b806e7875 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:32.135885 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:INSERT=2,keystone:DELETE=13,keystone:SELECT=85 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:32.164253 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 899/1801] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:32.172347 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b96f59-7fb5-4fb2-abcf-0923ad6be4fb None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:32.172676 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b96f59-7fb5-4fb2-abcf-0923ad6be4fb None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:32.172953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b96f59-7fb5-4fb2-abcf-0923ad6be4fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:32.200335 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-06b96f59-7fb5-4fb2-abcf-0923ad6be4fb None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:32.201624 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-06b96f59-7fb5-4fb2-abcf-0923ad6be4fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:32.208550 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:DELETE=11,keystone:SELECT=87,keystone:INSERT=2 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:32.220101 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 900/1802] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:32.221498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b96f59-7fb5-4fb2-abcf-0923ad6be4fb None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:32.223978 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-06b96f59-7fb5-4fb2-abcf-0923ad6be4fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:32.227129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879c1089-e463-4506-9a0b-7be5d0436e00 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:32.233140 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3ed65b-1ef3-4de9-9930-e40f67f9fa9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:32.234100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3ed65b-1ef3-4de9-9930-e40f67f9fa9d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:32.234526 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3ed65b-1ef3-4de9-9930-e40f67f9fa9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:32.234933 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3ed65b-1ef3-4de9-9930-e40f67f9fa9d admin admin] PATH_INFO: `/v3/projects/a6722e737b85493db74fd654f295f79d` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:32.236207 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3ed65b-1ef3-4de9-9930-e40f67f9fa9d admin admin] RBAC: Authorizing `identity:delete_project(project_id=a6722e737b85493db74fd654f295f79d)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:32.238140 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3ed65b-1ef3-4de9-9930-e40f67f9fa9d admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:32.246638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b96f59-7fb5-4fb2-abcf-0923ad6be4fb None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:32.252651 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaf46e7-3577-4a70-b31a-2e6f3c857052 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:32.253354 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaf46e7-3577-4a70-b31a-2e6f3c857052 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:32.253697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaf46e7-3577-4a70-b31a-2e6f3c857052 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:32.254068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaf46e7-3577-4a70-b31a-2e6f3c857052 admin admin] PATH_INFO: `/v3/projects/776f7ceb79e74aa3b1416863ddd4e049` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:32.255766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfaf46e7-3577-4a70-b31a-2e6f3c857052 admin admin] RBAC: Authorizing `identity:delete_project(project_id=776f7ceb79e74aa3b1416863ddd4e049)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:32.258042 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfaf46e7-3577-4a70-b31a-2e6f3c857052 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:32.299639 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-fb3ed65b-1ef3-4de9-9930-e40f67f9fa9d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a6722e737b85493db74fd654f295f79d', 'request_id': 'req-fb3ed65b-1ef3-4de9-9930-e40f67f9fa9d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:32.306919 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-fb3ed65b-1ef3-4de9-9930-e40f67f9fa9d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a6722e737b85493db74fd654f295f79d', 'request_id': 'req-fb3ed65b-1ef3-4de9-9930-e40f67f9fa9d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:32.314452 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-fb3ed65b-1ef3-4de9-9930-e40f67f9fa9d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a6722e737b85493db74fd654f295f79d', 'request_id': 'req-fb3ed65b-1ef3-4de9-9930-e40f67f9fa9d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:32.316243 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-bfaf46e7-3577-4a70-b31a-2e6f3c857052 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '776f7ceb79e74aa3b1416863ddd4e049', 'request_id': 'req-bfaf46e7-3577-4a70-b31a-2e6f3c857052'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:32.324611 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-bfaf46e7-3577-4a70-b31a-2e6f3c857052 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '776f7ceb79e74aa3b1416863ddd4e049', 'request_id': 'req-bfaf46e7-3577-4a70-b31a-2e6f3c857052'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:32.332711 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-bfaf46e7-3577-4a70-b31a-2e6f3c857052 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '776f7ceb79e74aa3b1416863ddd4e049', 'request_id': 'req-bfaf46e7-3577-4a70-b31a-2e6f3c857052'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:32.334352 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-fb3ed65b-1ef3-4de9-9930-e40f67f9fa9d admin admin] The token cache is being invalidate because project a6722e737b85493db74fd654f295f79d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:32.334715 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-fb3ed65b-1ef3-4de9-9930-e40f67f9fa9d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:32.335437 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-fb3ed65b-1ef3-4de9-9930-e40f67f9fa9d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:32.337640 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 903/1803] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:32 2022] DELETE /identity/v3/projects/a6722e737b85493db74fd654f295f79d => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:32.338424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3ed65b-1ef3-4de9-9930-e40f67f9fa9d admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:32.340170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-fb3ed65b-1ef3-4de9-9930-e40f67f9fa9d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:32.350595 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-bfaf46e7-3577-4a70-b31a-2e6f3c857052 admin admin] The token cache is being invalidate because project 776f7ceb79e74aa3b1416863ddd4e049 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:32.351100 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-bfaf46e7-3577-4a70-b31a-2e6f3c857052 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:32.352010 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-bfaf46e7-3577-4a70-b31a-2e6f3c857052 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:32.353837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 901/1804] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:32 2022] DELETE /identity/v3/projects/776f7ceb79e74aa3b1416863ddd4e049 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:32.357482 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91039abc-153b-423a-a517-63a7ec16cf7f None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:32.357482 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91039abc-153b-423a-a517-63a7ec16cf7f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:32.357482 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91039abc-153b-423a-a517-63a7ec16cf7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:32.378369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-91039abc-153b-423a-a517-63a7ec16cf7f None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:32.379283 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-91039abc-153b-423a-a517-63a7ec16cf7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:32.441393 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3ed65b-1ef3-4de9-9930-e40f67f9fa9d admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:32.445577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 902/1805] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:32.446233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc65e2f5-d2e4-492c-ad05-2c7fab0a39ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:32.446966 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc65e2f5-d2e4-492c-ad05-2c7fab0a39ac admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:32.447063 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfafb04-282e-4128-aee4-ed90780e0f8b None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:32.447281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfafb04-282e-4128-aee4-ed90780e0f8b None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:32.447505 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc65e2f5-d2e4-492c-ad05-2c7fab0a39ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:32.447617 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfafb04-282e-4128-aee4-ed90780e0f8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:32.447956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc65e2f5-d2e4-492c-ad05-2c7fab0a39ac admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:32.448875 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc65e2f5-d2e4-492c-ad05-2c7fab0a39ac admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:20:32.449122 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc65e2f5-d2e4-492c-ad05-2c7fab0a39ac admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:32.451131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc65e2f5-d2e4-492c-ad05-2c7fab0a39ac admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:32.464049 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 904/1806] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:20:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:20:32.465164 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc65e2f5-d2e4-492c-ad05-2c7fab0a39ac admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:32.467005 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-fc65e2f5-d2e4-492c-ad05-2c7fab0a39ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:32.467124 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-bdfafb04-282e-4128-aee4-ed90780e0f8b None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:32.468298 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-bdfafb04-282e-4128-aee4-ed90780e0f8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:32.476062 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 903/1807] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:32.477477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ba4cd2-d1fe-438a-ac8b-1591e8c91819 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:32.477898 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ba4cd2-d1fe-438a-ac8b-1591e8c91819 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:32.478409 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ba4cd2-d1fe-438a-ac8b-1591e8c91819 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:32.480390 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc65e2f5-d2e4-492c-ad05-2c7fab0a39ac admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:32.485791 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2429ec87-5419-48bd-bf79-5b8e35c251ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:32.486453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2429ec87-5419-48bd-bf79-5b8e35c251ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:32.486736 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2429ec87-5419-48bd-bf79-5b8e35c251ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:32.487014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2429ec87-5419-48bd-bf79-5b8e35c251ba admin admin] PATH_INFO: `/v3/projects/dcfbd01e12e548719c49888950d8cb56` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:32.488192 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2429ec87-5419-48bd-bf79-5b8e35c251ba admin admin] RBAC: Authorizing `identity:delete_project(project_id=dcfbd01e12e548719c49888950d8cb56)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:32.490159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2429ec87-5419-48bd-bf79-5b8e35c251ba admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:32.499884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-d8ba4cd2-d1fe-438a-ac8b-1591e8c91819 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:32.501003 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d8ba4cd2-d1fe-438a-ac8b-1591e8c91819 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:32.509036 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 904/1808] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:32.509822 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ba4cd2-d1fe-438a-ac8b-1591e8c91819 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:32.511009 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d8ba4cd2-d1fe-438a-ac8b-1591e8c91819 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:32.522351 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ba4cd2-d1fe-438a-ac8b-1591e8c91819 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:32.527034 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac627e94-1148-45b9-b356-095ac97b4934 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:32.527884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac627e94-1148-45b9-b356-095ac97b4934 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:32.528194 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac627e94-1148-45b9-b356-095ac97b4934 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:32.528430 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac627e94-1148-45b9-b356-095ac97b4934 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:32.529058 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac627e94-1148-45b9-b356-095ac97b4934 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:32.531057 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac627e94-1148-45b9-b356-095ac97b4934 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:32.544554 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-2429ec87-5419-48bd-bf79-5b8e35c251ba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dcfbd01e12e548719c49888950d8cb56', 'request_id': 'req-2429ec87-5419-48bd-bf79-5b8e35c251ba'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:32.552481 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-2429ec87-5419-48bd-bf79-5b8e35c251ba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dcfbd01e12e548719c49888950d8cb56', 'request_id': 'req-2429ec87-5419-48bd-bf79-5b8e35c251ba'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:32.559571 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-2429ec87-5419-48bd-bf79-5b8e35c251ba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dcfbd01e12e548719c49888950d8cb56', 'request_id': 'req-2429ec87-5419-48bd-bf79-5b8e35c251ba'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:32.562273 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 905/1809] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:32 2022] POST /identity/v3/projects => generated 393 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:32.563426 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54c80a2-4da0-4491-80f2-5534fb641dac None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:32.563715 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54c80a2-4da0-4491-80f2-5534fb641dac None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:32.563998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54c80a2-4da0-4491-80f2-5534fb641dac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:32.569212 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:32.575064 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-2429ec87-5419-48bd-bf79-5b8e35c251ba admin admin] The token cache is being invalidate because project dcfbd01e12e548719c49888950d8cb56 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:32.575668 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-2429ec87-5419-48bd-bf79-5b8e35c251ba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:32.576600 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-2429ec87-5419-48bd-bf79-5b8e35c251ba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:32.578301 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 905/1810] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:32 2022] DELETE /identity/v3/projects/dcfbd01e12e548719c49888950d8cb56 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:32.579175 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2429ec87-5419-48bd-bf79-5b8e35c251ba admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:32.580884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-2429ec87-5419-48bd-bf79-5b8e35c251ba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:32.585903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-a54c80a2-4da0-4491-80f2-5534fb641dac None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:32.587175 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a54c80a2-4da0-4491-80f2-5534fb641dac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:32.617119 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:32.651332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2429ec87-5419-48bd-bf79-5b8e35c251ba admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:32.655821 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 906/1811] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:32.655821 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54c80a2-4da0-4491-80f2-5534fb641dac None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:32.655821 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a54c80a2-4da0-4491-80f2-5534fb641dac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:32.659741 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2094a03-2d29-4580-9f6c-32f3657adff1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:32.661028 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2094a03-2d29-4580-9f6c-32f3657adff1 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:32.661028 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2094a03-2d29-4580-9f6c-32f3657adff1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:32.661028 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2094a03-2d29-4580-9f6c-32f3657adff1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:32.661544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2094a03-2d29-4580-9f6c-32f3657adff1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:20:32.661963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2094a03-2d29-4580-9f6c-32f3657adff1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:32.663552 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2094a03-2d29-4580-9f6c-32f3657adff1 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:32.666742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54c80a2-4da0-4491-80f2-5534fb641dac None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:32.672164 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e39d21f-709f-4bbb-87fb-e34a3f386550 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:32.672596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e39d21f-709f-4bbb-87fb-e34a3f386550 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:32.672886 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e39d21f-709f-4bbb-87fb-e34a3f386550 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:32.673139 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e39d21f-709f-4bbb-87fb-e34a3f386550 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:32.673787 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e39d21f-709f-4bbb-87fb-e34a3f386550 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:32.675696 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e39d21f-709f-4bbb-87fb-e34a3f386550 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:32.677612 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 906/1812] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:20:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:20:32.681616 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2094a03-2d29-4580-9f6c-32f3657adff1 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:32.683321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c2094a03-2d29-4580-9f6c-32f3657adff1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:32.698164 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2094a03-2d29-4580-9f6c-32f3657adff1 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:32.700224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 907/1813] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:32 2022] POST /identity/v3/users => generated 297 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:32.701222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e39d21f-709f-4bbb-87fb-e34a3f386550 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:32.703234 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3c63c4-dbba-4d5e-834e-8ba79ca91721 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:32.703873 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3c63c4-dbba-4d5e-834e-8ba79ca91721 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:32.704185 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3c63c4-dbba-4d5e-834e-8ba79ca91721 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:32.704464 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3c63c4-dbba-4d5e-834e-8ba79ca91721 admin admin] PATH_INFO: `/v3/users/da7a4bc732154f728255fbb3b7b7ec83` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:32.705604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3c63c4-dbba-4d5e-834e-8ba79ca91721 admin admin] RBAC: Authorizing `identity:delete_user(user_id=da7a4bc732154f728255fbb3b7b7ec83)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:32.708418 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3c63c4-dbba-4d5e-834e-8ba79ca91721 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:32.711837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e39d21f-709f-4bbb-87fb-e34a3f386550 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:32.716623 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cae4b3-db41-4b83-bd32-59c42fad5b02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:32.717269 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cae4b3-db41-4b83-bd32-59c42fad5b02 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:32.718292 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cae4b3-db41-4b83-bd32-59c42fad5b02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:32.718292 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cae4b3-db41-4b83-bd32-59c42fad5b02 admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:32.718579 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5cae4b3-db41-4b83-bd32-59c42fad5b02 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:20:32.718935 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5cae4b3-db41-4b83-bd32-59c42fad5b02 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:32.720533 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5cae4b3-db41-4b83-bd32-59c42fad5b02 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:32.741199 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 908/1814] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:20:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:20:32.742164 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cae4b3-db41-4b83-bd32-59c42fad5b02 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:32.763980 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cae4b3-db41-4b83-bd32-59c42fad5b02 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:32.771675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9c0fbc-bfe9-4a66-9ba3-cb00ef9ad033 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:32.772498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9c0fbc-bfe9-4a66-9ba3-cb00ef9ad033 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:32.772927 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9c0fbc-bfe9-4a66-9ba3-cb00ef9ad033 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:32.773815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9c0fbc-bfe9-4a66-9ba3-cb00ef9ad033 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:32.774736 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa9c0fbc-bfe9-4a66-9ba3-cb00ef9ad033 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:32.777399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa9c0fbc-bfe9-4a66-9ba3-cb00ef9ad033 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:32.810097 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-ca3c63c4-dbba-4d5e-834e-8ba79ca91721 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'da7a4bc732154f728255fbb3b7b7ec83'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:32.817909 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 909/1815] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:32 2022] POST /identity/v3/projects => generated 387 bytes in 76 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:32.818812 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-ca3c63c4-dbba-4d5e-834e-8ba79ca91721 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'da7a4bc732154f728255fbb3b7b7ec83'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:32.819257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9c0fbc-bfe9-4a66-9ba3-cb00ef9ad033 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:32.819874 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-ca3c63c4-dbba-4d5e-834e-8ba79ca91721 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'da7a4bc732154f728255fbb3b7b7ec83'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:32.826277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-fa9c0fbc-bfe9-4a66-9ba3-cb00ef9ad033 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:32.830641 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-ca3c63c4-dbba-4d5e-834e-8ba79ca91721 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'da7a4bc732154f728255fbb3b7b7ec83'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:32.847316 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-ca3c63c4-dbba-4d5e-834e-8ba79ca91721 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'da7a4bc732154f728255fbb3b7b7ec83'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:32.854259 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 907/1816] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:32 2022] DELETE /identity/v3/users/da7a4bc732154f728255fbb3b7b7ec83 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:32.855194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3c63c4-dbba-4d5e-834e-8ba79ca91721 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:32.856823 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ca3c63c4-dbba-4d5e-834e-8ba79ca91721 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:32.943419 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9c0fbc-bfe9-4a66-9ba3-cb00ef9ad033 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:32.946772 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3c63c4-dbba-4d5e-834e-8ba79ca91721 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:32.948847 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c14270-8428-40d6-a312-eef22ea6b25a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:32.949489 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c14270-8428-40d6-a312-eef22ea6b25a admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:32.949788 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c14270-8428-40d6-a312-eef22ea6b25a admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:32.950064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c14270-8428-40d6-a312-eef22ea6b25a admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:32.950710 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c14270-8428-40d6-a312-eef22ea6b25a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:32.953909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf95657-deee-4bf6-9c9f-53eb79f43639 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:32.954278 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf95657-deee-4bf6-9c9f-53eb79f43639 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:32.954704 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf95657-deee-4bf6-9c9f-53eb79f43639 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:32.955042 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf95657-deee-4bf6-9c9f-53eb79f43639 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:32.955751 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf95657-deee-4bf6-9c9f-53eb79f43639 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:32.957721 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf95657-deee-4bf6-9c9f-53eb79f43639 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:32.965354 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c14270-8428-40d6-a312-eef22ea6b25a admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:32.970260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:32.986313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 910/1817] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:32 2022] GET /identity/v3/roles => generated 1420 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:32.987469 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c14270-8428-40d6-a312-eef22ea6b25a admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:32.989729 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-51c14270-8428-40d6-a312-eef22ea6b25a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:33.000969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 908/1818] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:32 2022] POST /identity/v3/projects => generated 369 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:33.002034 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf95657-deee-4bf6-9c9f-53eb79f43639 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:33.009475 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c14270-8428-40d6-a312-eef22ea6b25a admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:33.098136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf95657-deee-4bf6-9c9f-53eb79f43639 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:33.103565 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0dce03-cd90-42dc-91f9-d6156551b5d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:33.104052 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0dce03-cd90-42dc-91f9-d6156551b5d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:33.104332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0dce03-cd90-42dc-91f9-d6156551b5d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:33.104607 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0dce03-cd90-42dc-91f9-d6156551b5d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:33.105257 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0dce03-cd90-42dc-91f9-d6156551b5d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:33.107002 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0dce03-cd90-42dc-91f9-d6156551b5d4 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:33.119547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 909/1819] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:33 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:33.120621 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0dce03-cd90-42dc-91f9-d6156551b5d4 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:33.126735 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee60adb-32c1-4cc3-98fd-8c52d1178f2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:33.127449 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee60adb-32c1-4cc3-98fd-8c52d1178f2c admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:33.127921 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee60adb-32c1-4cc3-98fd-8c52d1178f2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:33.128299 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee60adb-32c1-4cc3-98fd-8c52d1178f2c admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:33.129030 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee60adb-32c1-4cc3-98fd-8c52d1178f2c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:33.131358 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee60adb-32c1-4cc3-98fd-8c52d1178f2c admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:33.134332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0dce03-cd90-42dc-91f9-d6156551b5d4 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:33.142446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2153b78-d19d-4c6d-852c-38cc10362a81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:33.143385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2153b78-d19d-4c6d-852c-38cc10362a81 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:33.143703 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2153b78-d19d-4c6d-852c-38cc10362a81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:33.143983 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2153b78-d19d-4c6d-852c-38cc10362a81 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:33.144623 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2153b78-d19d-4c6d-852c-38cc10362a81 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:33.146549 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2153b78-d19d-4c6d-852c-38cc10362a81 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:33.168788 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 911/1820] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:32 2022] POST /identity/v3/users => generated 294 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:33.169768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee60adb-32c1-4cc3-98fd-8c52d1178f2c admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:33.178497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 910/1821] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:33 2022] POST /identity/v3/users => generated 285 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:33.179510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2153b78-d19d-4c6d-852c-38cc10362a81 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:33.193967 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2153b78-d19d-4c6d-852c-38cc10362a81 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:33.195827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee60adb-32c1-4cc3-98fd-8c52d1178f2c admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:33.200003 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b97d251-8dbd-4398-a08f-3c8fb4023879 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:33.200719 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b97d251-8dbd-4398-a08f-3c8fb4023879 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:33.200860 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0286c4f-317f-480e-b12b-4a28c24f6aa2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:33.200999 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b97d251-8dbd-4398-a08f-3c8fb4023879 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:33.201551 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0286c4f-317f-480e-b12b-4a28c24f6aa2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:33.201684 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b97d251-8dbd-4398-a08f-3c8fb4023879 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:33.201903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0286c4f-317f-480e-b12b-4a28c24f6aa2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:33.202697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0286c4f-317f-480e-b12b-4a28c24f6aa2 admin admin] PATH_INFO: `/v3/projects/1ff813b3184047249b2ad5ab9d6743f7/users/2ca715dd3543456daa09c9bbc9d21349/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:33.202910 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b97d251-8dbd-4398-a08f-3c8fb4023879 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:33.205319 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b97d251-8dbd-4398-a08f-3c8fb4023879 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:33.208185 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:33.226934 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 911/1822] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:33 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:33.227803 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b97d251-8dbd-4398-a08f-3c8fb4023879 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:33.235027 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0286c4f-317f-480e-b12b-4a28c24f6aa2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ff813b3184047249b2ad5ab9d6743f7, user_id=2ca715dd3543456daa09c9bbc9d21349, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:33.237703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0286c4f-317f-480e-b12b-4a28c24f6aa2 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:33.245265 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b97d251-8dbd-4398-a08f-3c8fb4023879 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:33.251398 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ddd951c-348e-42df-a528-52b637fc68ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:33.252127 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddd951c-348e-42df-a528-52b637fc68ca admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:33.252504 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddd951c-348e-42df-a528-52b637fc68ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:33.252951 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddd951c-348e-42df-a528-52b637fc68ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:33.254021 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddd951c-348e-42df-a528-52b637fc68ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:33.255777 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddd951c-348e-42df-a528-52b637fc68ca admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:33.266474 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 912/1823] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:33 2022] PUT /identity/v3/projects/1ff813b3184047249b2ad5ab9d6743f7/users/2ca715dd3543456daa09c9bbc9d21349/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:33.269790 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 912/1824] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:33 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:33.269790 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e18c21-f399-47ea-b04f-64762fe6c953 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:33.269790 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e18c21-f399-47ea-b04f-64762fe6c953 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:33.270179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e18c21-f399-47ea-b04f-64762fe6c953 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:33.270633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0286c4f-317f-480e-b12b-4a28c24f6aa2 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:33.326006 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-85e18c21-f399-47ea-b04f-64762fe6c953 None None] MFA Rules not processed for user `2ca715dd3543456daa09c9bbc9d21349`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:33.327848 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-85e18c21-f399-47ea-b04f-64762fe6c953 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:33.351367 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0286c4f-317f-480e-b12b-4a28c24f6aa2 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:33.356038 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf36179d-5ab1-4d2f-9cb1-f5f6755af7bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:33.356640 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf36179d-5ab1-4d2f-9cb1-f5f6755af7bd admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:33.356923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf36179d-5ab1-4d2f-9cb1-f5f6755af7bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:33.357225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf36179d-5ab1-4d2f-9cb1-f5f6755af7bd admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:33.357919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf36179d-5ab1-4d2f-9cb1-f5f6755af7bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:33.359531 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf36179d-5ab1-4d2f-9cb1-f5f6755af7bd admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:33.372720 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 913/1825] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:33 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:33.373575 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf36179d-5ab1-4d2f-9cb1-f5f6755af7bd admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:33.383098 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf36179d-5ab1-4d2f-9cb1-f5f6755af7bd admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:33.387755 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05efa050-dc61-4889-ad39-0aa6818f5c32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:33.388403 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05efa050-dc61-4889-ad39-0aa6818f5c32 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:33.388859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05efa050-dc61-4889-ad39-0aa6818f5c32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:33.389273 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05efa050-dc61-4889-ad39-0aa6818f5c32 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:33.390516 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05efa050-dc61-4889-ad39-0aa6818f5c32 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:33.392190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05efa050-dc61-4889-ad39-0aa6818f5c32 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:33.402993 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 913/1826] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:33 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:33.403861 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e18c21-f399-47ea-b04f-64762fe6c953 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:33.404608 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 914/1827] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:33 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:33.405860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-85e18c21-f399-47ea-b04f-64762fe6c953 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:33.406753 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05efa050-dc61-4889-ad39-0aa6818f5c32 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:33.411995 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-05efa050-dc61-4889-ad39-0aa6818f5c32 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:33.423363 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e18c21-f399-47ea-b04f-64762fe6c953 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:33.428302 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f59838-f016-43aa-afbf-1458823e128d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:33.428988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f59838-f016-43aa-afbf-1458823e128d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:33.429430 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f59838-f016-43aa-afbf-1458823e128d admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:33.430209 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f59838-f016-43aa-afbf-1458823e128d admin admin] PATH_INFO: `/v3/projects/d4d4abd3268e446fb4a1867b2ab17227` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:33.431434 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f59838-f016-43aa-afbf-1458823e128d admin admin] RBAC: Authorizing `identity:delete_project(project_id=d4d4abd3268e446fb4a1867b2ab17227)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:33.432453 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05efa050-dc61-4889-ad39-0aa6818f5c32 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:33.434830 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f59838-f016-43aa-afbf-1458823e128d admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:33.437369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b02f536-c1aa-4caf-8ea6-2b85d120b481 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:33.438036 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b02f536-c1aa-4caf-8ea6-2b85d120b481 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:33.438315 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b02f536-c1aa-4caf-8ea6-2b85d120b481 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:33.438590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b02f536-c1aa-4caf-8ea6-2b85d120b481 admin admin] PATH_INFO: `/v3/users/3fbf83b5c00e46dd83975789618071e8` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:33.439813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b02f536-c1aa-4caf-8ea6-2b85d120b481 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3fbf83b5c00e46dd83975789618071e8)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:33.441609 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b02f536-c1aa-4caf-8ea6-2b85d120b481 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:33.474366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=94,keystone:INSERT=4,keystone:DELETE=2 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:33.492015 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-56f59838-f016-43aa-afbf-1458823e128d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd4d4abd3268e446fb4a1867b2ab17227', 'request_id': 'req-56f59838-f016-43aa-afbf-1458823e128d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:33.496351 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6b02f536-c1aa-4caf-8ea6-2b85d120b481 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3fbf83b5c00e46dd83975789618071e8'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:33.498935 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-56f59838-f016-43aa-afbf-1458823e128d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd4d4abd3268e446fb4a1867b2ab17227', 'request_id': 'req-56f59838-f016-43aa-afbf-1458823e128d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:33.504554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6b02f536-c1aa-4caf-8ea6-2b85d120b481 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3fbf83b5c00e46dd83975789618071e8'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:33.505209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6b02f536-c1aa-4caf-8ea6-2b85d120b481 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3fbf83b5c00e46dd83975789618071e8'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:33.507338 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-56f59838-f016-43aa-afbf-1458823e128d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd4d4abd3268e446fb4a1867b2ab17227', 'request_id': 'req-56f59838-f016-43aa-afbf-1458823e128d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:33.514324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6b02f536-c1aa-4caf-8ea6-2b85d120b481 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3fbf83b5c00e46dd83975789618071e8'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:33.522621 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-56f59838-f016-43aa-afbf-1458823e128d admin admin] The token cache is being invalidate because project d4d4abd3268e446fb4a1867b2ab17227 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:33.522920 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-56f59838-f016-43aa-afbf-1458823e128d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:33.523592 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-56f59838-f016-43aa-afbf-1458823e128d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:33.525306 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 914/1828] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:33 2022] DELETE /identity/v3/projects/d4d4abd3268e446fb4a1867b2ab17227 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:33.530811 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f59838-f016-43aa-afbf-1458823e128d admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:33.530811 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-56f59838-f016-43aa-afbf-1458823e128d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:33.534922 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6b02f536-c1aa-4caf-8ea6-2b85d120b481 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3fbf83b5c00e46dd83975789618071e8'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:33.542231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 915/1829] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:33 2022] DELETE /identity/v3/users/3fbf83b5c00e46dd83975789618071e8 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:33.543173 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b02f536-c1aa-4caf-8ea6-2b85d120b481 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:33.544567 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6b02f536-c1aa-4caf-8ea6-2b85d120b481 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:33.593697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:33.611764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f59838-f016-43aa-afbf-1458823e128d admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:33.614576 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b02f536-c1aa-4caf-8ea6-2b85d120b481 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:33.616849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb6a77d-184a-4072-9020-1dca54b41c1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:33.617516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb6a77d-184a-4072-9020-1dca54b41c1d admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:33.617831 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb6a77d-184a-4072-9020-1dca54b41c1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:33.618111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb6a77d-184a-4072-9020-1dca54b41c1d admin admin] PATH_INFO: `/v3/projects/5d736ae2114748c3801d1ae02603fe79/users/205fe88d1b7c4f1da25a1e68e277efa9/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:33.619181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742ab991-1ed3-4c6c-b10d-23aaca0e9d7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:33.619837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742ab991-1ed3-4c6c-b10d-23aaca0e9d7e admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:33.620238 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742ab991-1ed3-4c6c-b10d-23aaca0e9d7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:33.620401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742ab991-1ed3-4c6c-b10d-23aaca0e9d7e admin admin] PATH_INFO: `/v3/projects/23cf7efa73b14cab92137fda57640001/users/fea4a3a9230f416082f438b9f9e481ad/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:33.633527 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb6a77d-184a-4072-9020-1dca54b41c1d admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d736ae2114748c3801d1ae02603fe79, user_id=205fe88d1b7c4f1da25a1e68e277efa9, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:33.635301 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742ab991-1ed3-4c6c-b10d-23aaca0e9d7e admin admin] RBAC: Authorizing `identity:create_grant(project_id=23cf7efa73b14cab92137fda57640001, user_id=fea4a3a9230f416082f438b9f9e481ad, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:33.636243 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb6a77d-184a-4072-9020-1dca54b41c1d admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:33.637933 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742ab991-1ed3-4c6c-b10d-23aaca0e9d7e admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:33.649615 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 915/1830] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:33 2022] PUT /identity/v3/projects/5d736ae2114748c3801d1ae02603fe79/users/205fe88d1b7c4f1da25a1e68e277efa9/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:33.650500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb6a77d-184a-4072-9020-1dca54b41c1d admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:33.661297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 916/1831] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:33 2022] PUT /identity/v3/projects/23cf7efa73b14cab92137fda57640001/users/fea4a3a9230f416082f438b9f9e481ad/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:33.661485 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-4fb6a77d-184a-4072-9020-1dca54b41c1d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:33.661581 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e42fcc9-35cc-4066-8dd8-38b1af2da692 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:33.661581 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e42fcc9-35cc-4066-8dd8-38b1af2da692 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:33.661581 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e42fcc9-35cc-4066-8dd8-38b1af2da692 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:33.677504 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-2e42fcc9-35cc-4066-8dd8-38b1af2da692 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:33.678796 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-2e42fcc9-35cc-4066-8dd8-38b1af2da692 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:33.721685 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb6a77d-184a-4072-9020-1dca54b41c1d admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:33.736538 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 917/1832] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:33.737953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942159ef-3186-47ed-9a3c-65e28f3061f8 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:33.738238 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942159ef-3186-47ed-9a3c-65e28f3061f8 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:33.738881 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942159ef-3186-47ed-9a3c-65e28f3061f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:33.758787 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-942159ef-3186-47ed-9a3c-65e28f3061f8 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:33.759900 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-942159ef-3186-47ed-9a3c-65e28f3061f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:33.767980 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:33.768472 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 918/1833] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:33.772923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83dcdc7-7152-4885-b25a-c356802b8a06 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:33.773210 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83dcdc7-7152-4885-b25a-c356802b8a06 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:33.773653 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83dcdc7-7152-4885-b25a-c356802b8a06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:33.783071 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238e6cf9-7154-4d5b-b14b-b70db50cb0cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:33.783849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238e6cf9-7154-4d5b-b14b-b70db50cb0cb admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:33.784284 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238e6cf9-7154-4d5b-b14b-b70db50cb0cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:33.784767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238e6cf9-7154-4d5b-b14b-b70db50cb0cb admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:33.785405 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238e6cf9-7154-4d5b-b14b-b70db50cb0cb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:33.787325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238e6cf9-7154-4d5b-b14b-b70db50cb0cb admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:33.801614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-c83dcdc7-7152-4885-b25a-c356802b8a06 None None] MFA Rules not processed for user `205fe88d1b7c4f1da25a1e68e277efa9`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:33.803109 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c83dcdc7-7152-4885-b25a-c356802b8a06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:33.817463 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 916/1834] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:33 2022] POST /identity/v3/projects => generated 393 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:33.817463 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8db6f71-490e-4cba-a2d1-409d40367292 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:33.817463 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8db6f71-490e-4cba-a2d1-409d40367292 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:33.817463 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8db6f71-490e-4cba-a2d1-409d40367292 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:33.845050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-e8db6f71-490e-4cba-a2d1-409d40367292 None None] MFA Rules not processed for user `fea4a3a9230f416082f438b9f9e481ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:33.846293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-e8db6f71-490e-4cba-a2d1-409d40367292 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:33.888825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 919/1835] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:33 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:33.891568 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15196939-f3ac-45b9-8213-d26eb8af8c50 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:33.892016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15196939-f3ac-45b9-8213-d26eb8af8c50 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:33.892293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15196939-f3ac-45b9-8213-d26eb8af8c50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:33.916225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-15196939-f3ac-45b9-8213-d26eb8af8c50 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:33.917473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-15196939-f3ac-45b9-8213-d26eb8af8c50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:33.928310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 917/1836] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:33 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:33.928310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8db6f71-490e-4cba-a2d1-409d40367292 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:33.929542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-e8db6f71-490e-4cba-a2d1-409d40367292 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:33.966438 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:33.986097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 920/1837] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:33.987078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15196939-f3ac-45b9-8213-d26eb8af8c50 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:33.995435 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8db6f71-490e-4cba-a2d1-409d40367292 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:33.999634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a62d49-6c97-4248-887b-5664b1ec693e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:34.000942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a62d49-6c97-4248-887b-5664b1ec693e service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:34.000942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a62d49-6c97-4248-887b-5664b1ec693e service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:34.000942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a62d49-6c97-4248-887b-5664b1ec693e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:34.002592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15196939-f3ac-45b9-8213-d26eb8af8c50 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:34.008314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fdfe51-1714-4838-b806-f167a8979495 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:34.009851 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fdfe51-1714-4838-b806-f167a8979495 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:34.010416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fdfe51-1714-4838-b806-f167a8979495 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:34.010955 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fdfe51-1714-4838-b806-f167a8979495 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:34.011853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fdfe51-1714-4838-b806-f167a8979495 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:34.012101 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a62d49-6c97-4248-887b-5664b1ec693e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:34.013997 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a62d49-6c97-4248-887b-5664b1ec693e service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:34.014115 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fdfe51-1714-4838-b806-f167a8979495 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:34.026353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 918/1838] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:34.027200 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a62d49-6c97-4248-887b-5664b1ec693e service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:34.041873 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a62d49-6c97-4248-887b-5664b1ec693e service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:34.050654 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 921/1839] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:33 2022] POST /identity/v3/users => generated 303 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:34.051535 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fdfe51-1714-4838-b806-f167a8979495 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:34.054034 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-b6fdfe51-1714-4838-b806-f167a8979495 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:34.054441 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511fb5ac-26fd-4804-9a88-5eb5285cde80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:34.055793 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511fb5ac-26fd-4804-9a88-5eb5285cde80 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:34.056215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511fb5ac-26fd-4804-9a88-5eb5285cde80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:34.056678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511fb5ac-26fd-4804-9a88-5eb5285cde80 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:34.057728 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511fb5ac-26fd-4804-9a88-5eb5285cde80 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:34.060718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511fb5ac-26fd-4804-9a88-5eb5285cde80 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:34.072269 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fdfe51-1714-4838-b806-f167a8979495 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:34.078570 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caac60ad-01fa-4bea-81c3-b9618a965b53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:34.079186 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caac60ad-01fa-4bea-81c3-b9618a965b53 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:34.079879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caac60ad-01fa-4bea-81c3-b9618a965b53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:34.080153 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caac60ad-01fa-4bea-81c3-b9618a965b53 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:34.080814 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caac60ad-01fa-4bea-81c3-b9618a965b53 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:34.083409 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caac60ad-01fa-4bea-81c3-b9618a965b53 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:34.112676 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 919/1840] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:34 2022] POST /identity/v3/projects => generated 387 bytes in 86 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:34.113679 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511fb5ac-26fd-4804-9a88-5eb5285cde80 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:34.139201 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 922/1841] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:34 2022] POST /identity/v3/projects => generated 367 bytes in 85 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:34.139201 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caac60ad-01fa-4bea-81c3-b9618a965b53 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:34.179835 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:34.201363 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511fb5ac-26fd-4804-9a88-5eb5285cde80 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:34.242610 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caac60ad-01fa-4bea-81c3-b9618a965b53 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:34.247713 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc319a41-d7b2-44d1-969d-c9ee3b59c9c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:34.248517 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc319a41-d7b2-44d1-969d-c9ee3b59c9c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:34.250681 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc319a41-d7b2-44d1-969d-c9ee3b59c9c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:34.250681 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc319a41-d7b2-44d1-969d-c9ee3b59c9c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:34.251399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc319a41-d7b2-44d1-969d-c9ee3b59c9c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:34.253120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc319a41-d7b2-44d1-969d-c9ee3b59c9c7 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:34.268991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 923/1842] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:34 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:34.268991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc319a41-d7b2-44d1-969d-c9ee3b59c9c7 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:34.279439 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc319a41-d7b2-44d1-969d-c9ee3b59c9c7 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:34.281624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db016c07-c061-4c07-9fde-3bdf1b0da3ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:34.282321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db016c07-c061-4c07-9fde-3bdf1b0da3ea admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:34.282730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db016c07-c061-4c07-9fde-3bdf1b0da3ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:34.283038 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db016c07-c061-4c07-9fde-3bdf1b0da3ea admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:34.283778 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db016c07-c061-4c07-9fde-3bdf1b0da3ea admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:20:34.284271 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db016c07-c061-4c07-9fde-3bdf1b0da3ea admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:34.286191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db016c07-c061-4c07-9fde-3bdf1b0da3ea admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:34.286605 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b469682-7518-4492-959d-a8aecee1a9fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:34.287272 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b469682-7518-4492-959d-a8aecee1a9fd admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:34.287562 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b469682-7518-4492-959d-a8aecee1a9fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:34.288133 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b469682-7518-4492-959d-a8aecee1a9fd admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:34.288465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b469682-7518-4492-959d-a8aecee1a9fd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:34.290451 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b469682-7518-4492-959d-a8aecee1a9fd admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:34.305543 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 920/1843] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:20:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:20:34.305543 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db016c07-c061-4c07-9fde-3bdf1b0da3ea admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:34.314853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 924/1844] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:34 2022] POST /identity/v3/users => generated 300 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:34.315869 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b469682-7518-4492-959d-a8aecee1a9fd admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:34.316455 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db016c07-c061-4c07-9fde-3bdf1b0da3ea admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:34.321476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3165f5-c063-47c1-9f36-4e273e780ebd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:34.322318 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3165f5-c063-47c1-9f36-4e273e780ebd admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:34.322608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3165f5-c063-47c1-9f36-4e273e780ebd admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:34.322882 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3165f5-c063-47c1-9f36-4e273e780ebd admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:34.323513 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3165f5-c063-47c1-9f36-4e273e780ebd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:34.325408 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3165f5-c063-47c1-9f36-4e273e780ebd admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:34.328459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b469682-7518-4492-959d-a8aecee1a9fd admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:34.333722 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9d4954-f5ee-4597-a429-5417d4ac5e47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:34.334560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9d4954-f5ee-4597-a429-5417d4ac5e47 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:34.335119 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9d4954-f5ee-4597-a429-5417d4ac5e47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:34.335449 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9d4954-f5ee-4597-a429-5417d4ac5e47 admin admin] PATH_INFO: `/v3/projects/0ec09a0648bd421981a633352221b9dd/users/a4b986d177014d749891075e7692d902/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:34.356118 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9d4954-f5ee-4597-a429-5417d4ac5e47 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ec09a0648bd421981a633352221b9dd, user_id=a4b986d177014d749891075e7692d902, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:34.360258 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9d4954-f5ee-4597-a429-5417d4ac5e47 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:34.375242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 921/1845] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:34 2022] POST /identity/v3/users => generated 290 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:34.376033 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3165f5-c063-47c1-9f36-4e273e780ebd admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:34.381628 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 925/1846] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:34 2022] PUT /identity/v3/projects/0ec09a0648bd421981a633352221b9dd/users/a4b986d177014d749891075e7692d902/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:34.382678 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9d4954-f5ee-4597-a429-5417d4ac5e47 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:34.396940 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3165f5-c063-47c1-9f36-4e273e780ebd admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:34.482442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9d4954-f5ee-4597-a429-5417d4ac5e47 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:34.488592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7ae6ac-9c1b-4f0d-8d71-4bdf3c054071 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:34.489415 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7ae6ac-9c1b-4f0d-8d71-4bdf3c054071 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:34.489874 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7ae6ac-9c1b-4f0d-8d71-4bdf3c054071 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:34.490289 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7ae6ac-9c1b-4f0d-8d71-4bdf3c054071 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:34.491065 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7ae6ac-9c1b-4f0d-8d71-4bdf3c054071 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:34.493432 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7ae6ac-9c1b-4f0d-8d71-4bdf3c054071 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:34.502160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:34.509209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 926/1847] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:34 2022] GET /identity/v3/roles => generated 1420 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:34.510669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7ae6ac-9c1b-4f0d-8d71-4bdf3c054071 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:34.514969 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-cc7ae6ac-9c1b-4f0d-8d71-4bdf3c054071 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:34.518502 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06076f2-95b5-41cc-adec-e551cda455ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:34.519180 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06076f2-95b5-41cc-adec-e551cda455ce admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:34.519399 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06076f2-95b5-41cc-adec-e551cda455ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:34.519672 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06076f2-95b5-41cc-adec-e551cda455ce admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:34.520431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06076f2-95b5-41cc-adec-e551cda455ce admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:34.522667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06076f2-95b5-41cc-adec-e551cda455ce admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:34.536657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7ae6ac-9c1b-4f0d-8d71-4bdf3c054071 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:34.552020 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659a6417-a7d1-4ce1-8ec3-baf166807117 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:34.553020 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659a6417-a7d1-4ce1-8ec3-baf166807117 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:34.553020 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659a6417-a7d1-4ce1-8ec3-baf166807117 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:34.553219 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659a6417-a7d1-4ce1-8ec3-baf166807117 admin admin] PATH_INFO: `/v3/projects/de1f5d9b493942d088cbc5e3925d94c3` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:34.557206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659a6417-a7d1-4ce1-8ec3-baf166807117 admin admin] RBAC: Authorizing `identity:delete_project(project_id=de1f5d9b493942d088cbc5e3925d94c3)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:34.559257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659a6417-a7d1-4ce1-8ec3-baf166807117 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:34.571638 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 922/1848] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:34 2022] POST /identity/v3/projects => generated 385 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:34.572657 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06076f2-95b5-41cc-adec-e551cda455ce admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:34.653915 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:34.671006 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06076f2-95b5-41cc-adec-e551cda455ce admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:34.677432 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-659a6417-a7d1-4ce1-8ec3-baf166807117 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'de1f5d9b493942d088cbc5e3925d94c3', 'request_id': 'req-659a6417-a7d1-4ce1-8ec3-baf166807117'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:34.686033 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-659a6417-a7d1-4ce1-8ec3-baf166807117 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'de1f5d9b493942d088cbc5e3925d94c3', 'request_id': 'req-659a6417-a7d1-4ce1-8ec3-baf166807117'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:34.694724 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-659a6417-a7d1-4ce1-8ec3-baf166807117 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'de1f5d9b493942d088cbc5e3925d94c3', 'request_id': 'req-659a6417-a7d1-4ce1-8ec3-baf166807117'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:34.714933 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-659a6417-a7d1-4ce1-8ec3-baf166807117 admin admin] The token cache is being invalidate because project de1f5d9b493942d088cbc5e3925d94c3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:34.715351 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-659a6417-a7d1-4ce1-8ec3-baf166807117 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:34.716431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-659a6417-a7d1-4ce1-8ec3-baf166807117 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:34.723754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 927/1849] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:34 2022] DELETE /identity/v3/projects/de1f5d9b493942d088cbc5e3925d94c3 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:34.724807 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659a6417-a7d1-4ce1-8ec3-baf166807117 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:34.726768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-659a6417-a7d1-4ce1-8ec3-baf166807117 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:34.749446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d342380d-635b-4752-8ec2-55457912e49b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:34.750177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d342380d-635b-4752-8ec2-55457912e49b admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:34.750573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d342380d-635b-4752-8ec2-55457912e49b admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:34.750910 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d342380d-635b-4752-8ec2-55457912e49b admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:34.752694 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d342380d-635b-4752-8ec2-55457912e49b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:34.754911 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d342380d-635b-4752-8ec2-55457912e49b admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:34.768860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 923/1850] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:34 2022] GET /identity/v3/roles => generated 1420 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:34.769793 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d342380d-635b-4752-8ec2-55457912e49b admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:34.772571 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-d342380d-635b-4752-8ec2-55457912e49b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:34.787530 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d342380d-635b-4752-8ec2-55457912e49b admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:34.793854 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d07ff16-d00e-4c7a-b1a3-fc1046227057 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:34.795102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d07ff16-d00e-4c7a-b1a3-fc1046227057 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:34.795102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d07ff16-d00e-4c7a-b1a3-fc1046227057 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:34.795385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d07ff16-d00e-4c7a-b1a3-fc1046227057 admin admin] PATH_INFO: `/v3/projects/ea4892e5b25b4462ad734616243f81fa/users/bb6d6c13513544d8adb3c39e80147091/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:34.809168 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659a6417-a7d1-4ce1-8ec3-baf166807117 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:34.815365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d07ff16-d00e-4c7a-b1a3-fc1046227057 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea4892e5b25b4462ad734616243f81fa, user_id=bb6d6c13513544d8adb3c39e80147091, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:34.815871 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469b17ae-e3e9-4224-bd91-9befe93b74c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:34.816492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469b17ae-e3e9-4224-bd91-9befe93b74c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:34.816765 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469b17ae-e3e9-4224-bd91-9befe93b74c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:34.817184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469b17ae-e3e9-4224-bd91-9befe93b74c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:34.818204 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469b17ae-e3e9-4224-bd91-9befe93b74c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:34.818327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d07ff16-d00e-4c7a-b1a3-fc1046227057 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:34.819489 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469b17ae-e3e9-4224-bd91-9befe93b74c7 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:34.831379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 928/1851] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:34 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:34.832371 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469b17ae-e3e9-4224-bd91-9befe93b74c7 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:34.834058 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-469b17ae-e3e9-4224-bd91-9befe93b74c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:34.834910 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 924/1852] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:34 2022] PUT /identity/v3/projects/ea4892e5b25b4462ad734616243f81fa/users/bb6d6c13513544d8adb3c39e80147091/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:34.838421 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d07ff16-d00e-4c7a-b1a3-fc1046227057 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:34.840180 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-4d07ff16-d00e-4c7a-b1a3-fc1046227057 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:34.848267 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469b17ae-e3e9-4224-bd91-9befe93b74c7 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:34.924924 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d07ff16-d00e-4c7a-b1a3-fc1046227057 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:34.926115 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1655fe95-d797-4bdd-8416-78385197838b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:34.926730 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1655fe95-d797-4bdd-8416-78385197838b admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:34.927008 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1655fe95-d797-4bdd-8416-78385197838b admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:34.927282 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1655fe95-d797-4bdd-8416-78385197838b admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:34.927912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1655fe95-d797-4bdd-8416-78385197838b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:34.929721 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c661da16-41b9-4211-b774-8ae527654ea3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:34.929905 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1655fe95-d797-4bdd-8416-78385197838b admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:34.930388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c661da16-41b9-4211-b774-8ae527654ea3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:34.930669 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c661da16-41b9-4211-b774-8ae527654ea3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:34.930944 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c661da16-41b9-4211-b774-8ae527654ea3 admin admin] PATH_INFO: `/v3/users/2502f6864180491faac5ec99e95dba5e` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:34.932114 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c661da16-41b9-4211-b774-8ae527654ea3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2502f6864180491faac5ec99e95dba5e)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:34.934121 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c661da16-41b9-4211-b774-8ae527654ea3 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:34.952871 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 929/1853] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:34 2022] POST /identity/v3/users => generated 293 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:34.953672 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1655fe95-d797-4bdd-8416-78385197838b admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:34.955303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-1655fe95-d797-4bdd-8416-78385197838b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:34.966840 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1655fe95-d797-4bdd-8416-78385197838b admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:34.971643 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853f5d0c-e93d-4255-bc2e-0d74d55bdd20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:34.972300 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853f5d0c-e93d-4255-bc2e-0d74d55bdd20 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:34.972573 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853f5d0c-e93d-4255-bc2e-0d74d55bdd20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:34.972862 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853f5d0c-e93d-4255-bc2e-0d74d55bdd20 admin admin] PATH_INFO: `/v3/projects/a25926354d504c84a348bbe506b83b3a/users/4663f0df277a40f496432e95ea3445c7/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:34.975980 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:34.984966 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-c661da16-41b9-4211-b774-8ae527654ea3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2502f6864180491faac5ec99e95dba5e'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:34.991037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-853f5d0c-e93d-4255-bc2e-0d74d55bdd20 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a25926354d504c84a348bbe506b83b3a, user_id=4663f0df277a40f496432e95ea3445c7, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:34.993830 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-853f5d0c-e93d-4255-bc2e-0d74d55bdd20 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:34.994493 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-c661da16-41b9-4211-b774-8ae527654ea3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2502f6864180491faac5ec99e95dba5e'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:34.995290 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-c661da16-41b9-4211-b774-8ae527654ea3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2502f6864180491faac5ec99e95dba5e'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:35.004684 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-c661da16-41b9-4211-b774-8ae527654ea3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2502f6864180491faac5ec99e95dba5e'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:35.008690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 930/1854] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:34 2022] PUT /identity/v3/projects/a25926354d504c84a348bbe506b83b3a/users/4663f0df277a40f496432e95ea3445c7/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:35.012286 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853f5d0c-e93d-4255-bc2e-0d74d55bdd20 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:35.014215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:INSERT=2,keystone:DELETE=9 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:35.015178 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-853f5d0c-e93d-4255-bc2e-0d74d55bdd20 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:35.019694 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-c661da16-41b9-4211-b774-8ae527654ea3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2502f6864180491faac5ec99e95dba5e'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:35.027417 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 925/1855] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:34 2022] DELETE /identity/v3/users/2502f6864180491faac5ec99e95dba5e => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:35.028463 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c661da16-41b9-4211-b774-8ae527654ea3 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:35.042286 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c661da16-41b9-4211-b774-8ae527654ea3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:35.110416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853f5d0c-e93d-4255-bc2e-0d74d55bdd20 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:35.146291 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c661da16-41b9-4211-b774-8ae527654ea3 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:35.154204 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ce3cf9-46f3-4742-9548-89fa519d70bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:35.161333 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ce3cf9-46f3-4742-9548-89fa519d70bd admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:35.161624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ce3cf9-46f3-4742-9548-89fa519d70bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:35.162158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ce3cf9-46f3-4742-9548-89fa519d70bd admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:35.162996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ce3cf9-46f3-4742-9548-89fa519d70bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:35.164612 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ce3cf9-46f3-4742-9548-89fa519d70bd admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:35.185198 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 926/1856] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:35 2022] GET /identity/v3/roles => generated 1420 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:35.187164 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ce3cf9-46f3-4742-9548-89fa519d70bd admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:35.191038 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-f5ce3cf9-46f3-4742-9548-89fa519d70bd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:35.207642 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ce3cf9-46f3-4742-9548-89fa519d70bd admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:35.215334 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b685fa-d7c6-4431-849f-fe80ad1c788b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:35.218338 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b685fa-d7c6-4431-849f-fe80ad1c788b admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:35.218338 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b685fa-d7c6-4431-849f-fe80ad1c788b admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:35.218338 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b685fa-d7c6-4431-849f-fe80ad1c788b admin admin] PATH_INFO: `/v3/projects/0ec09a0648bd421981a633352221b9dd/users/a4b986d177014d749891075e7692d902/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:35.219649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae28eecf-a047-4eb9-ad69-a0219e8a8e8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:35.220356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae28eecf-a047-4eb9-ad69-a0219e8a8e8a admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:35.220726 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae28eecf-a047-4eb9-ad69-a0219e8a8e8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:35.221085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae28eecf-a047-4eb9-ad69-a0219e8a8e8a admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:35.221772 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae28eecf-a047-4eb9-ad69-a0219e8a8e8a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:35.223893 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae28eecf-a047-4eb9-ad69-a0219e8a8e8a admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:35.232065 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b685fa-d7c6-4431-849f-fe80ad1c788b admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ec09a0648bd421981a633352221b9dd, user_id=a4b986d177014d749891075e7692d902, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:35.236486 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b685fa-d7c6-4431-849f-fe80ad1c788b admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:35.241840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 927/1857] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:35 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:35.242943 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae28eecf-a047-4eb9-ad69-a0219e8a8e8a admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:35.247525 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ae28eecf-a047-4eb9-ad69-a0219e8a8e8a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:35.255639 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 931/1858] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:35 2022] PUT /identity/v3/projects/0ec09a0648bd421981a633352221b9dd/users/a4b986d177014d749891075e7692d902/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:35.265587 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae28eecf-a047-4eb9-ad69-a0219e8a8e8a admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:35.266791 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b685fa-d7c6-4431-849f-fe80ad1c788b admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:35.371296 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b685fa-d7c6-4431-849f-fe80ad1c788b admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:35.376668 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535ce700-53a7-4b6c-a3be-7900cd0f2c9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:35.377379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535ce700-53a7-4b6c-a3be-7900cd0f2c9a admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:35.377633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535ce700-53a7-4b6c-a3be-7900cd0f2c9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:35.377942 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535ce700-53a7-4b6c-a3be-7900cd0f2c9a admin admin] PATH_INFO: `/v3/projects/ea4892e5b25b4462ad734616243f81fa/users/bb6d6c13513544d8adb3c39e80147091/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:35.379998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535ce700-53a7-4b6c-a3be-7900cd0f2c9a admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea4892e5b25b4462ad734616243f81fa, user_id=bb6d6c13513544d8adb3c39e80147091, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:35.380129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229ec5a7-1bfb-4082-8fb0-2c608df903ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:35.380853 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229ec5a7-1bfb-4082-8fb0-2c608df903ef admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:35.381137 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229ec5a7-1bfb-4082-8fb0-2c608df903ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:35.381406 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229ec5a7-1bfb-4082-8fb0-2c608df903ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:35.382083 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229ec5a7-1bfb-4082-8fb0-2c608df903ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:35.382919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535ce700-53a7-4b6c-a3be-7900cd0f2c9a admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:35.383778 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229ec5a7-1bfb-4082-8fb0-2c608df903ef admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:35.409777 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 928/1859] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:35 2022] GET /identity/v3/roles => generated 1420 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:35.409777 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229ec5a7-1bfb-4082-8fb0-2c608df903ef admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:35.410058 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 932/1860] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:35 2022] PUT /identity/v3/projects/ea4892e5b25b4462ad734616243f81fa/users/bb6d6c13513544d8adb3c39e80147091/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:35.410058 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df3e941-28bc-42d3-8dd4-30494a4117c4 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:35.410058 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df3e941-28bc-42d3-8dd4-30494a4117c4 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:35.410058 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df3e941-28bc-42d3-8dd4-30494a4117c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:35.424322 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229ec5a7-1bfb-4082-8fb0-2c608df903ef admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:35.446610 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-4df3e941-28bc-42d3-8dd4-30494a4117c4 None None] MFA Rules not processed for user `a4b986d177014d749891075e7692d902`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:35.448305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-4df3e941-28bc-42d3-8dd4-30494a4117c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:35.460253 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:35.493841 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236e555e-606b-4fe2-ab83-6e9148b476be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:35.494459 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236e555e-606b-4fe2-ab83-6e9148b476be admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:35.494739 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236e555e-606b-4fe2-ab83-6e9148b476be admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:35.495009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236e555e-606b-4fe2-ab83-6e9148b476be admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:35.496012 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236e555e-606b-4fe2-ab83-6e9148b476be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:35.497936 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236e555e-606b-4fe2-ab83-6e9148b476be admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:35.507967 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 929/1861] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:35 2022] GET /identity/v3/roles => generated 1420 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:35.508743 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236e555e-606b-4fe2-ab83-6e9148b476be admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:35.519193 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236e555e-606b-4fe2-ab83-6e9148b476be admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:35.523731 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bed5ed1-081d-4168-9fb4-26642790a1c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:35.524452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 933/1862] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:35 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:35.524592 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bed5ed1-081d-4168-9fb4-26642790a1c8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:35.524671 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bed5ed1-081d-4168-9fb4-26642790a1c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:35.524894 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bed5ed1-081d-4168-9fb4-26642790a1c8 admin admin] PATH_INFO: `/v3/projects/a25926354d504c84a348bbe506b83b3a/users/4663f0df277a40f496432e95ea3445c7/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:35.525537 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae51bd81-4e5a-42a0-ad66-66f583251921 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:35.525964 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae51bd81-4e5a-42a0-ad66-66f583251921 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:35.526244 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae51bd81-4e5a-42a0-ad66-66f583251921 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:35.526888 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bed5ed1-081d-4168-9fb4-26642790a1c8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a25926354d504c84a348bbe506b83b3a, user_id=4663f0df277a40f496432e95ea3445c7, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:35.530177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bed5ed1-081d-4168-9fb4-26642790a1c8 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:35.533812 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=98,keystone:DELETE=1,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:35.543757 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 930/1863] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:35 2022] PUT /identity/v3/projects/a25926354d504c84a348bbe506b83b3a/users/4663f0df277a40f496432e95ea3445c7/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:35.545148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bed5ed1-081d-4168-9fb4-26642790a1c8 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:35.551114 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0bed5ed1-081d-4168-9fb4-26642790a1c8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:35.561155 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-ae51bd81-4e5a-42a0-ad66-66f583251921 None None] MFA Rules not processed for user `bb6d6c13513544d8adb3c39e80147091`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:35.562364 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ae51bd81-4e5a-42a0-ad66-66f583251921 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:35.616758 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bed5ed1-081d-4168-9fb4-26642790a1c8 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:35.621561 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8211fe0d-87db-4da1-8eb7-3e1b84fffdb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:35.622268 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8211fe0d-87db-4da1-8eb7-3e1b84fffdb8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:35.622574 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8211fe0d-87db-4da1-8eb7-3e1b84fffdb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:35.622876 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8211fe0d-87db-4da1-8eb7-3e1b84fffdb8 admin admin] PATH_INFO: `/v3/projects/a7a58f1304fb4f4493d562f7e2ba707e` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:35.624070 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8211fe0d-87db-4da1-8eb7-3e1b84fffdb8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a7a58f1304fb4f4493d562f7e2ba707e)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:35.625996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8211fe0d-87db-4da1-8eb7-3e1b84fffdb8 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:35.629461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 934/1864] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:35 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:35.630392 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae51bd81-4e5a-42a0-ad66-66f583251921 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:35.641729 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae51bd81-4e5a-42a0-ad66-66f583251921 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:35.646762 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd6c9e3-725d-46bd-acb3-b4c4e0ba8246 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:35.647716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd6c9e3-725d-46bd-acb3-b4c4e0ba8246 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:35.648086 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd6c9e3-725d-46bd-acb3-b4c4e0ba8246 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:35.648498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd6c9e3-725d-46bd-acb3-b4c4e0ba8246 admin admin] PATH_INFO: `/v3/projects/17c0e187271d4aab948bd1d9395d35d3/users/0b5828afcc48467fa76b308eb8426f64/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:35.665794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd6c9e3-725d-46bd-acb3-b4c4e0ba8246 admin admin] RBAC: Authorizing `identity:create_grant(project_id=17c0e187271d4aab948bd1d9395d35d3, user_id=0b5828afcc48467fa76b308eb8426f64, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:35.668427 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd6c9e3-725d-46bd-acb3-b4c4e0ba8246 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:35.679889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8211fe0d-87db-4da1-8eb7-3e1b84fffdb8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a7a58f1304fb4f4493d562f7e2ba707e', 'request_id': 'req-8211fe0d-87db-4da1-8eb7-3e1b84fffdb8'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:35.694213 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 935/1865] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:35 2022] PUT /identity/v3/projects/17c0e187271d4aab948bd1d9395d35d3/users/0b5828afcc48467fa76b308eb8426f64/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:35.694213 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd6c9e3-725d-46bd-acb3-b4c4e0ba8246 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:35.694213 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ebd6c9e3-725d-46bd-acb3-b4c4e0ba8246 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:35.697260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8211fe0d-87db-4da1-8eb7-3e1b84fffdb8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a7a58f1304fb4f4493d562f7e2ba707e', 'request_id': 'req-8211fe0d-87db-4da1-8eb7-3e1b84fffdb8'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:35.705064 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8211fe0d-87db-4da1-8eb7-3e1b84fffdb8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a7a58f1304fb4f4493d562f7e2ba707e', 'request_id': 'req-8211fe0d-87db-4da1-8eb7-3e1b84fffdb8'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:35.720152 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8211fe0d-87db-4da1-8eb7-3e1b84fffdb8 admin admin] The token cache is being invalidate because project a7a58f1304fb4f4493d562f7e2ba707e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:35.720448 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8211fe0d-87db-4da1-8eb7-3e1b84fffdb8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:35.721210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8211fe0d-87db-4da1-8eb7-3e1b84fffdb8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:35.723390 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 931/1866] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:35 2022] DELETE /identity/v3/projects/a7a58f1304fb4f4493d562f7e2ba707e => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:35.724744 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c30800-2752-4c49-aea3-5f8137a60d5a None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:35.725029 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c30800-2752-4c49-aea3-5f8137a60d5a None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:35.725303 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c30800-2752-4c49-aea3-5f8137a60d5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:35.758813 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-96c30800-2752-4c49-aea3-5f8137a60d5a None None] MFA Rules not processed for user `4663f0df277a40f496432e95ea3445c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:35.760246 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-96c30800-2752-4c49-aea3-5f8137a60d5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:35.768101 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd6c9e3-725d-46bd-acb3-b4c4e0ba8246 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:35.771765 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcffa8ca-f67e-4e66-bde3-f2bb98481e1e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:35.772370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcffa8ca-f67e-4e66-bde3-f2bb98481e1e service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:35.772641 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcffa8ca-f67e-4e66-bde3-f2bb98481e1e service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:35.772907 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcffa8ca-f67e-4e66-bde3-f2bb98481e1e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:35.774651 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-bcffa8ca-f67e-4e66-bde3-f2bb98481e1e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:35.797461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:35.837321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 932/1867] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:35 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:35.838119 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c30800-2752-4c49-aea3-5f8137a60d5a None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:35.839345 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-96c30800-2752-4c49-aea3-5f8137a60d5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:35.848355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcffa8ca-f67e-4e66-bde3-f2bb98481e1e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:35.849826 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcffa8ca-f67e-4e66-bde3-f2bb98481e1e service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:35.850952 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c30800-2752-4c49-aea3-5f8137a60d5a None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:35.855005 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02004ca-c99a-47ac-a8e6-ad58fe9ee2ab service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:35.855167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02004ca-c99a-47ac-a8e6-ad58fe9ee2ab service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:35.855442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02004ca-c99a-47ac-a8e6-ad58fe9ee2ab service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:35.855709 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02004ca-c99a-47ac-a8e6-ad58fe9ee2ab service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:35.857347 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-e02004ca-c99a-47ac-a8e6-ad58fe9ee2ab service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:35.859294 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 936/1868] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:35 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:35.860549 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94624d1-c66a-409f-a3a9-363b56d2ba8c None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:35.860784 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94624d1-c66a-409f-a3a9-363b56d2ba8c None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:35.861050 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94624d1-c66a-409f-a3a9-363b56d2ba8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:35.866899 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:35.890531 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-d94624d1-c66a-409f-a3a9-363b56d2ba8c None None] MFA Rules not processed for user `0b5828afcc48467fa76b308eb8426f64`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:35.892030 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d94624d1-c66a-409f-a3a9-363b56d2ba8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:35.927899 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e02004ca-c99a-47ac-a8e6-ad58fe9ee2ab service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:35.929656 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e02004ca-c99a-47ac-a8e6-ad58fe9ee2ab service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:35.943137 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 933/1869] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:35 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:35.944567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13c1dff-1c1c-44d1-937d-59dc0a0a21ce None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:35.944827 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13c1dff-1c1c-44d1-937d-59dc0a0a21ce None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:35.945157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13c1dff-1c1c-44d1-937d-59dc0a0a21ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:35.972343 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-c13c1dff-1c1c-44d1-937d-59dc0a0a21ce None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:35.973575 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c13c1dff-1c1c-44d1-937d-59dc0a0a21ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:35.975251 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 937/1870] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:35 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:35.976344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94624d1-c66a-409f-a3a9-363b56d2ba8c None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:35.977955 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d94624d1-c66a-409f-a3a9-363b56d2ba8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:35.991184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94624d1-c66a-409f-a3a9-363b56d2ba8c None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:35.995064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3a134e-a450-4e36-9dd9-f392d0dedf9f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:35.995706 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3a134e-a450-4e36-9dd9-f392d0dedf9f service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:35.995995 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3a134e-a450-4e36-9dd9-f392d0dedf9f service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:35.996265 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3a134e-a450-4e36-9dd9-f392d0dedf9f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:36.012560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3a134e-a450-4e36-9dd9-f392d0dedf9f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:36.015715 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3a134e-a450-4e36-9dd9-f392d0dedf9f service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:36.029631 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 938/1871] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:35 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:36.071966 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 934/1872] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:36.074261 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a138d773-263c-470f-8c45-58e682f0ee08 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:36.074791 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a138d773-263c-470f-8c45-58e682f0ee08 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:36.075237 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a138d773-263c-470f-8c45-58e682f0ee08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:36.107283 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13c1dff-1c1c-44d1-937d-59dc0a0a21ce None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:36.119737 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-a138d773-263c-470f-8c45-58e682f0ee08 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:36.121954 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a138d773-263c-470f-8c45-58e682f0ee08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:36.129583 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13c1dff-1c1c-44d1-937d-59dc0a0a21ce None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:36.146199 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6df54e-fb32-4f73-80cd-403152f5d890 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:36.148641 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6df54e-fb32-4f73-80cd-403152f5d890 service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:36.149209 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6df54e-fb32-4f73-80cd-403152f5d890 service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:36.150208 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6df54e-fb32-4f73-80cd-403152f5d890 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:36.153086 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-bf6df54e-fb32-4f73-80cd-403152f5d890 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:36.159711 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 939/1873] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:36.159711 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a138d773-263c-470f-8c45-58e682f0ee08 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:36.180873 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a138d773-263c-470f-8c45-58e682f0ee08 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:36.184888 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d406f53a-4121-45da-96de-f4fd5fdaf983 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:36.185532 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d406f53a-4121-45da-96de-f4fd5fdaf983 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:36.185845 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d406f53a-4121-45da-96de-f4fd5fdaf983 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:36.186112 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d406f53a-4121-45da-96de-f4fd5fdaf983 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:36.195936 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d406f53a-4121-45da-96de-f4fd5fdaf983 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:36.197386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d406f53a-4121-45da-96de-f4fd5fdaf983 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:36.206366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 940/1874] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:36 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:36.207238 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d406f53a-4121-45da-96de-f4fd5fdaf983 service cinder] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:36.218303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d406f53a-4121-45da-96de-f4fd5fdaf983 service cinder] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:36.221225 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf6df54e-fb32-4f73-80cd-403152f5d890 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:36.223032 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf6df54e-fb32-4f73-80cd-403152f5d890 service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:36.223727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace82c14-b246-4fce-befd-5934cfca793e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:36.224321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace82c14-b246-4fce-befd-5934cfca793e admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:36.224600 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace82c14-b246-4fce-befd-5934cfca793e admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:36.224860 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace82c14-b246-4fce-befd-5934cfca793e admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:36.225494 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace82c14-b246-4fce-befd-5934cfca793e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:20:36.225820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace82c14-b246-4fce-befd-5934cfca793e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:36.227415 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace82c14-b246-4fce-befd-5934cfca793e admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:36.233112 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 935/1875] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:36 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:20:36.245779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 941/1876] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:20:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:20:36.246654 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6df54e-fb32-4f73-80cd-403152f5d890 service cinder] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:36.258079 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6df54e-fb32-4f73-80cd-403152f5d890 service cinder] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:36.262972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9afee1-8aa3-4811-bbac-1234ae3d8a49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:36.263578 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9afee1-8aa3-4811-bbac-1234ae3d8a49 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:36.263860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9afee1-8aa3-4811-bbac-1234ae3d8a49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:36.264205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9afee1-8aa3-4811-bbac-1234ae3d8a49 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:36.264826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9afee1-8aa3-4811-bbac-1234ae3d8a49 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:36.266725 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9afee1-8aa3-4811-bbac-1234ae3d8a49 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:36.297866 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 936/1877] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:36 2022] POST /identity/v3/projects => generated 381 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:36.302001 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace82c14-b246-4fce-befd-5934cfca793e admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:36.312534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace82c14-b246-4fce-befd-5934cfca793e admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:36.394586 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc333d80-2741-47c0-97af-26292b0114d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:36.395206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc333d80-2741-47c0-97af-26292b0114d8 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:36.395483 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc333d80-2741-47c0-97af-26292b0114d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:36.395798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc333d80-2741-47c0-97af-26292b0114d8 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:36.397485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc333d80-2741-47c0-97af-26292b0114d8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:36.401542 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc333d80-2741-47c0-97af-26292b0114d8 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:36.413608 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:36.438373 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 942/1878] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:36 2022] POST /identity/v3/users => generated 291 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:36.438887 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9afee1-8aa3-4811-bbac-1234ae3d8a49 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:36.451786 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9afee1-8aa3-4811-bbac-1234ae3d8a49 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:36.462719 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3199c85-4986-4f1e-875a-36ee3d9184b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:36.463205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3199c85-4986-4f1e-875a-36ee3d9184b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:36.463577 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3199c85-4986-4f1e-875a-36ee3d9184b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:36.463945 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3199c85-4986-4f1e-875a-36ee3d9184b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:36.470591 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3199c85-4986-4f1e-875a-36ee3d9184b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:36.477275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3199c85-4986-4f1e-875a-36ee3d9184b2 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:36.503258 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 937/1879] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:36 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:36.504900 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc333d80-2741-47c0-97af-26292b0114d8 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:36.519775 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc333d80-2741-47c0-97af-26292b0114d8 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:36.530017 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d04da3-997f-4fe3-9372-cf720ee6ed3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:36.531285 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d04da3-997f-4fe3-9372-cf720ee6ed3b admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:36.531656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d04da3-997f-4fe3-9372-cf720ee6ed3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:36.532021 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d04da3-997f-4fe3-9372-cf720ee6ed3b admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:36.533326 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d04da3-997f-4fe3-9372-cf720ee6ed3b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:36.534488 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d04da3-997f-4fe3-9372-cf720ee6ed3b admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:36.564018 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3199c85-4986-4f1e-875a-36ee3d9184b2 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:36.574925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 943/1880] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:36 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:36.578484 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3199c85-4986-4f1e-875a-36ee3d9184b2 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:36.586988 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d04da3-997f-4fe3-9372-cf720ee6ed3b admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:36.600753 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86306ce0-fa61-4283-bd93-6a4cf9f8f619 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:36.601565 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86306ce0-fa61-4283-bd93-6a4cf9f8f619 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:36.601972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86306ce0-fa61-4283-bd93-6a4cf9f8f619 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:36.602336 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86306ce0-fa61-4283-bd93-6a4cf9f8f619 admin admin] PATH_INFO: `/v3/projects/c5b4f63d1eb24b8eae0fe749cea44218/users/7d7cb00f4f2d42e1abc9202fcedb3e82/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:36.615399 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:36.637224 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86306ce0-fa61-4283-bd93-6a4cf9f8f619 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c5b4f63d1eb24b8eae0fe749cea44218, user_id=7d7cb00f4f2d42e1abc9202fcedb3e82, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:36.640396 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86306ce0-fa61-4283-bd93-6a4cf9f8f619 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:36.669654 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 938/1881] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:36 2022] PUT /identity/v3/projects/c5b4f63d1eb24b8eae0fe749cea44218/users/7d7cb00f4f2d42e1abc9202fcedb3e82/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:36.680443 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841baf35-7a2d-4457-83e3-8ad11baea0de None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:36.681737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841baf35-7a2d-4457-83e3-8ad11baea0de None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:36.682318 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841baf35-7a2d-4457-83e3-8ad11baea0de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:36.690460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d04da3-997f-4fe3-9372-cf720ee6ed3b admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:36.734339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-841baf35-7a2d-4457-83e3-8ad11baea0de None None] MFA Rules not processed for user `7d7cb00f4f2d42e1abc9202fcedb3e82`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:36.736144 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-841baf35-7a2d-4457-83e3-8ad11baea0de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:36.756335 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72600fe9-7cc9-447c-ac3e-b79a6fb1cab1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:36.756877 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72600fe9-7cc9-447c-ac3e-b79a6fb1cab1 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:36.757253 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72600fe9-7cc9-447c-ac3e-b79a6fb1cab1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:36.757526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72600fe9-7cc9-447c-ac3e-b79a6fb1cab1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:36.761422 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-72600fe9-7cc9-447c-ac3e-b79a6fb1cab1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:36.877457 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72600fe9-7cc9-447c-ac3e-b79a6fb1cab1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:36.879464 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72600fe9-7cc9-447c-ac3e-b79a6fb1cab1 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:36.894185 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 944/1882] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:36 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:36.897238 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72600fe9-7cc9-447c-ac3e-b79a6fb1cab1 service cinder] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:36.899302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-72600fe9-7cc9-447c-ac3e-b79a6fb1cab1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:36.916580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 939/1883] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:36 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:36.917815 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841baf35-7a2d-4457-83e3-8ad11baea0de None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:36.988496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:37.012737 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72600fe9-7cc9-447c-ac3e-b79a6fb1cab1 service cinder] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:37.026746 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3e0403-49b4-405d-8667-b28950d04dfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:37.028139 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3e0403-49b4-405d-8667-b28950d04dfe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:37.028426 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3e0403-49b4-405d-8667-b28950d04dfe admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:37.028949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3e0403-49b4-405d-8667-b28950d04dfe admin admin] PATH_INFO: `/v3/users/205fe88d1b7c4f1da25a1e68e277efa9` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:37.031229 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3e0403-49b4-405d-8667-b28950d04dfe admin admin] RBAC: Authorizing `identity:delete_user(user_id=205fe88d1b7c4f1da25a1e68e277efa9)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:37.034039 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3e0403-49b4-405d-8667-b28950d04dfe admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:37.052270 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841baf35-7a2d-4457-83e3-8ad11baea0de None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:37.063708 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a617a4c-33c9-4fb3-999f-2beda9c13615 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:37.064365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a617a4c-33c9-4fb3-999f-2beda9c13615 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:37.064719 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a617a4c-33c9-4fb3-999f-2beda9c13615 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:37.064994 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a617a4c-33c9-4fb3-999f-2beda9c13615 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:37.065778 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a617a4c-33c9-4fb3-999f-2beda9c13615 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:37.068087 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a617a4c-33c9-4fb3-999f-2beda9c13615 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:37.109676 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6c3e0403-49b4-405d-8667-b28950d04dfe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '205fe88d1b7c4f1da25a1e68e277efa9'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:37.114864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 940/1884] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:36 2022] POST /identity/v3/projects => generated 383 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:37.118606 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a1a316-9836-4bc1-9b02-39ee011da1bb None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:37.118899 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a1a316-9836-4bc1-9b02-39ee011da1bb None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:37.119178 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a1a316-9836-4bc1-9b02-39ee011da1bb None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:37.122640 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6c3e0403-49b4-405d-8667-b28950d04dfe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '205fe88d1b7c4f1da25a1e68e277efa9'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:37.122987 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 941/1885] 10.222.0.26 () {54 vars in 921 bytes} [Tue Sep 27 04:20:37 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:20:37.123675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6c3e0403-49b4-405d-8667-b28950d04dfe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '205fe88d1b7c4f1da25a1e68e277efa9'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:37.123793 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a1a316-9836-4bc1-9b02-39ee011da1bb None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:37.128375 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-b2a1a316-9836-4bc1-9b02-39ee011da1bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:37.136243 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6c3e0403-49b4-405d-8667-b28950d04dfe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '205fe88d1b7c4f1da25a1e68e277efa9'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:37.172125 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6c3e0403-49b4-405d-8667-b28950d04dfe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '205fe88d1b7c4f1da25a1e68e277efa9'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:37.182783 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 945/1886] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:36 2022] DELETE /identity/v3/users/205fe88d1b7c4f1da25a1e68e277efa9 => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:37.187107 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02fe1ef8-7b19-43eb-806a-0b613a0605da None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:37.188867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02fe1ef8-7b19-43eb-806a-0b613a0605da None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:37.189220 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02fe1ef8-7b19-43eb-806a-0b613a0605da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:37.220349 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-02fe1ef8-7b19-43eb-806a-0b613a0605da None None] MFA Rules not processed for user `a1586ee841cd44658e4d5feb25de7940`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:37.221850 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-02fe1ef8-7b19-43eb-806a-0b613a0605da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:37.300635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a1a316-9836-4bc1-9b02-39ee011da1bb None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:37.309727 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:37.346118 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 946/1887] 10.222.0.26 () {58 vars in 1018 bytes} [Tue Sep 27 04:20:37 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:37.348916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9867c065-e8aa-4d60-81fa-a55de50eba59 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:37.349302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9867c065-e8aa-4d60-81fa-a55de50eba59 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:37.349822 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9867c065-e8aa-4d60-81fa-a55de50eba59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:37.399354 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-9867c065-e8aa-4d60-81fa-a55de50eba59 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:37.403371 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-9867c065-e8aa-4d60-81fa-a55de50eba59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:37.408925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73948eb8-fe65-431a-8b47-9684cb550902 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:37.409799 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73948eb8-fe65-431a-8b47-9684cb550902 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:37.410239 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73948eb8-fe65-431a-8b47-9684cb550902 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:37.410638 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73948eb8-fe65-431a-8b47-9684cb550902 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:37.411516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73948eb8-fe65-431a-8b47-9684cb550902 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:37.414400 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73948eb8-fe65-431a-8b47-9684cb550902 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:37.426483 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 947/1888] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:37.430729 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9867c065-e8aa-4d60-81fa-a55de50eba59 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:37.436198 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-9867c065-e8aa-4d60-81fa-a55de50eba59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:37.457037 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 942/1889] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:37 2022] POST /identity/v3/users => generated 298 bytes in 331 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:37.457899 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73948eb8-fe65-431a-8b47-9684cb550902 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:37.459545 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-73948eb8-fe65-431a-8b47-9684cb550902 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:37.460828 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9867c065-e8aa-4d60-81fa-a55de50eba59 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:37.469728 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ba6c08-248d-4c75-8a4f-e03b101ddc0a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:37.470544 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ba6c08-248d-4c75-8a4f-e03b101ddc0a service nova] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:37.470841 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ba6c08-248d-4c75-8a4f-e03b101ddc0a service nova] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:37.471237 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ba6c08-248d-4c75-8a4f-e03b101ddc0a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:37.487194 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38ba6c08-248d-4c75-8a4f-e03b101ddc0a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:37.488838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38ba6c08-248d-4c75-8a4f-e03b101ddc0a service nova] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:37.502505 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 948/1890] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:37 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 72 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:20:37.504703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ba6c08-248d-4c75-8a4f-e03b101ddc0a service nova] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:37.517232 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ba6c08-248d-4c75-8a4f-e03b101ddc0a service nova] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:37.522555 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261f3c67-61f7-482c-aff7-2982681c705b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:37.523152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261f3c67-61f7-482c-aff7-2982681c705b admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:37.523423 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261f3c67-61f7-482c-aff7-2982681c705b admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:37.523692 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261f3c67-61f7-482c-aff7-2982681c705b admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:37.524332 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261f3c67-61f7-482c-aff7-2982681c705b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:37.526127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261f3c67-61f7-482c-aff7-2982681c705b admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:37.537431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 949/1891] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:37 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:37.542112 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261f3c67-61f7-482c-aff7-2982681c705b admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:37.554423 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261f3c67-61f7-482c-aff7-2982681c705b admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:37.557423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73948eb8-fe65-431a-8b47-9684cb550902 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:37.559927 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b048a16c-fbcb-41ad-80a0-802e3acfe7ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:37.560587 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b048a16c-fbcb-41ad-80a0-802e3acfe7ec admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:37.560869 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b048a16c-fbcb-41ad-80a0-802e3acfe7ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:37.561282 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b048a16c-fbcb-41ad-80a0-802e3acfe7ec admin admin] PATH_INFO: `/v3/projects/3a7201c9aa744077b931b70cdbc7295b/users/a853049913cb45f2b473c266d12f73ae/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:37.562450 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae12037c-abe5-4a82-9d96-8322a1bf5fa9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:37.563154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae12037c-abe5-4a82-9d96-8322a1bf5fa9 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:37.564859 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae12037c-abe5-4a82-9d96-8322a1bf5fa9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:37.565637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae12037c-abe5-4a82-9d96-8322a1bf5fa9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:37.579742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b048a16c-fbcb-41ad-80a0-802e3acfe7ec admin admin] RBAC: Authorizing `identity:create_grant(project_id=3a7201c9aa744077b931b70cdbc7295b, user_id=a853049913cb45f2b473c266d12f73ae, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:37.581319 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae12037c-abe5-4a82-9d96-8322a1bf5fa9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:37.582797 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b048a16c-fbcb-41ad-80a0-802e3acfe7ec admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:37.582938 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae12037c-abe5-4a82-9d96-8322a1bf5fa9 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:37.595481 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 943/1892] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:20:37.605256 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 950/1893] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:37 2022] PUT /identity/v3/projects/3a7201c9aa744077b931b70cdbc7295b/users/a853049913cb45f2b473c266d12f73ae/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:20:37.610128 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae12037c-abe5-4a82-9d96-8322a1bf5fa9 service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:37.620134 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae12037c-abe5-4a82-9d96-8322a1bf5fa9 service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:37.697452 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005dbf77-be85-4f08-a53d-08ae9702f0e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:37.698784 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005dbf77-be85-4f08-a53d-08ae9702f0e4 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:37.699078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005dbf77-be85-4f08-a53d-08ae9702f0e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:37.699355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005dbf77-be85-4f08-a53d-08ae9702f0e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:37.700679 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-005dbf77-be85-4f08-a53d-08ae9702f0e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:37.702372 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-005dbf77-be85-4f08-a53d-08ae9702f0e4 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:37.716093 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 944/1894] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:37 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:37.720309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b048a16c-fbcb-41ad-80a0-802e3acfe7ec admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:37.734487 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:37.741306 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b048a16c-fbcb-41ad-80a0-802e3acfe7ec admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:37.748718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700ca982-b4e9-4e88-b4bb-e5ecfce47b8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:37.750198 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700ca982-b4e9-4e88-b4bb-e5ecfce47b8e admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:37.750575 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700ca982-b4e9-4e88-b4bb-e5ecfce47b8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:37.750857 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700ca982-b4e9-4e88-b4bb-e5ecfce47b8e admin admin] PATH_INFO: `/v3/projects/3a7201c9aa744077b931b70cdbc7295b/users/a853049913cb45f2b473c266d12f73ae/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:37.753922 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700ca982-b4e9-4e88-b4bb-e5ecfce47b8e admin admin] RBAC: Authorizing `identity:create_grant(project_id=3a7201c9aa744077b931b70cdbc7295b, user_id=a853049913cb45f2b473c266d12f73ae, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:37.756705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700ca982-b4e9-4e88-b4bb-e5ecfce47b8e admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:37.781804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 951/1895] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:37 2022] PUT /identity/v3/projects/3a7201c9aa744077b931b70cdbc7295b/users/a853049913cb45f2b473c266d12f73ae/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:37.793698 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa8ef5c-e232-4e91-b160-9b3fdb8dff75 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:37.794088 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa8ef5c-e232-4e91-b160-9b3fdb8dff75 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:37.794532 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa8ef5c-e232-4e91-b160-9b3fdb8dff75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:37.829197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-baa8ef5c-e232-4e91-b160-9b3fdb8dff75 None None] MFA Rules not processed for user `a853049913cb45f2b473c266d12f73ae`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:37.830542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-baa8ef5c-e232-4e91-b160-9b3fdb8dff75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:37.870189 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:37.990532 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 945/1896] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:37 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:38.162192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700ca982-b4e9-4e88-b4bb-e5ecfce47b8e admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:38.163520 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-700ca982-b4e9-4e88-b4bb-e5ecfce47b8e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:38.180666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa8ef5c-e232-4e91-b160-9b3fdb8dff75 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:38.183222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-baa8ef5c-e232-4e91-b160-9b3fdb8dff75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:38.300518 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700ca982-b4e9-4e88-b4bb-e5ecfce47b8e admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:38.309272 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c53e2b-eb45-4a76-ab18-920117101f53 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:38.310319 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c53e2b-eb45-4a76-ab18-920117101f53 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:38.314141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c53e2b-eb45-4a76-ab18-920117101f53 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:38.314418 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c53e2b-eb45-4a76-ab18-920117101f53 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:38.320323 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa8ef5c-e232-4e91-b160-9b3fdb8dff75 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:38.325183 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244fb980-6d4c-4eea-94ce-24f60c735d97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:38.326019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244fb980-6d4c-4eea-94ce-24f60c735d97 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:38.326310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244fb980-6d4c-4eea-94ce-24f60c735d97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:38.326587 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244fb980-6d4c-4eea-94ce-24f60c735d97 admin admin] PATH_INFO: `/v3/projects/5d736ae2114748c3801d1ae02603fe79` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:38.327831 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244fb980-6d4c-4eea-94ce-24f60c735d97 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5d736ae2114748c3801d1ae02603fe79)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:38.329132 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c53e2b-eb45-4a76-ab18-920117101f53 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:38.329920 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244fb980-6d4c-4eea-94ce-24f60c735d97 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:38.331481 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c53e2b-eb45-4a76-ab18-920117101f53 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:38.345926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 952/1897] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:38 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:38.374993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c53e2b-eb45-4a76-ab18-920117101f53 service cinder] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:38.388076 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c53e2b-eb45-4a76-ab18-920117101f53 service cinder] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:38.394576 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-244fb980-6d4c-4eea-94ce-24f60c735d97 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5d736ae2114748c3801d1ae02603fe79', 'request_id': 'req-244fb980-6d4c-4eea-94ce-24f60c735d97'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:38.402994 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-244fb980-6d4c-4eea-94ce-24f60c735d97 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5d736ae2114748c3801d1ae02603fe79', 'request_id': 'req-244fb980-6d4c-4eea-94ce-24f60c735d97'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:38.430984 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-244fb980-6d4c-4eea-94ce-24f60c735d97 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5d736ae2114748c3801d1ae02603fe79', 'request_id': 'req-244fb980-6d4c-4eea-94ce-24f60c735d97'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:38.459646 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-244fb980-6d4c-4eea-94ce-24f60c735d97 admin admin] The token cache is being invalidate because project 5d736ae2114748c3801d1ae02603fe79 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:38.459949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-244fb980-6d4c-4eea-94ce-24f60c735d97 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:38.460658 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-244fb980-6d4c-4eea-94ce-24f60c735d97 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:38.464933 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 946/1898] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:38 2022] DELETE /identity/v3/projects/5d736ae2114748c3801d1ae02603fe79 => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:38.469854 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244fb980-6d4c-4eea-94ce-24f60c735d97 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:38.471671 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-244fb980-6d4c-4eea-94ce-24f60c735d97 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:38.504567 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267433e0-e1ba-4558-9f02-2fd4a91b0e44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:38.505780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267433e0-e1ba-4558-9f02-2fd4a91b0e44 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:38.506207 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267433e0-e1ba-4558-9f02-2fd4a91b0e44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:38.506582 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267433e0-e1ba-4558-9f02-2fd4a91b0e44 admin admin] PATH_INFO: `/v3/users/7d7cb00f4f2d42e1abc9202fcedb3e82` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:38.508213 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267433e0-e1ba-4558-9f02-2fd4a91b0e44 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7d7cb00f4f2d42e1abc9202fcedb3e82)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:38.512853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267433e0-e1ba-4558-9f02-2fd4a91b0e44 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:38.538745 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:38.564548 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244fb980-6d4c-4eea-94ce-24f60c735d97 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:38.572720 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7079732-3d27-48ec-99f9-492a63c49174 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:38.574886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7079732-3d27-48ec-99f9-492a63c49174 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:38.575314 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7079732-3d27-48ec-99f9-492a63c49174 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:38.576132 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7079732-3d27-48ec-99f9-492a63c49174 admin admin] PATH_INFO: `/v3/users/bb6d6c13513544d8adb3c39e80147091` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:38.578765 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7079732-3d27-48ec-99f9-492a63c49174 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bb6d6c13513544d8adb3c39e80147091)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:38.582092 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7079732-3d27-48ec-99f9-492a63c49174 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:38.615221 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-267433e0-e1ba-4558-9f02-2fd4a91b0e44 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7d7cb00f4f2d42e1abc9202fcedb3e82'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:38.627599 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-267433e0-e1ba-4558-9f02-2fd4a91b0e44 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7d7cb00f4f2d42e1abc9202fcedb3e82'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:38.627599 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-267433e0-e1ba-4558-9f02-2fd4a91b0e44 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7d7cb00f4f2d42e1abc9202fcedb3e82'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:38.644350 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-267433e0-e1ba-4558-9f02-2fd4a91b0e44 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7d7cb00f4f2d42e1abc9202fcedb3e82'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:38.665086 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a7079732-3d27-48ec-99f9-492a63c49174 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bb6d6c13513544d8adb3c39e80147091'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:38.666444 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-267433e0-e1ba-4558-9f02-2fd4a91b0e44 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7d7cb00f4f2d42e1abc9202fcedb3e82'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:38.679219 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a7079732-3d27-48ec-99f9-492a63c49174 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bb6d6c13513544d8adb3c39e80147091'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:38.679897 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a7079732-3d27-48ec-99f9-492a63c49174 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bb6d6c13513544d8adb3c39e80147091'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:38.680574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 953/1899] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:38 2022] DELETE /identity/v3/users/7d7cb00f4f2d42e1abc9202fcedb3e82 => generated 0 bytes in 306 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:38.686524 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6831484-3805-4e30-b0ba-e8ea0deb05d8 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:38.686864 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6831484-3805-4e30-b0ba-e8ea0deb05d8 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:38.687148 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6831484-3805-4e30-b0ba-e8ea0deb05d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:38.700449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a7079732-3d27-48ec-99f9-492a63c49174 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bb6d6c13513544d8adb3c39e80147091'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:38.713025 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-f6831484-3805-4e30-b0ba-e8ea0deb05d8 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:38.714236 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-f6831484-3805-4e30-b0ba-e8ea0deb05d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:38.723395 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a7079732-3d27-48ec-99f9-492a63c49174 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bb6d6c13513544d8adb3c39e80147091'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:38.726481 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:38.737702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 947/1900] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:38 2022] DELETE /identity/v3/users/bb6d6c13513544d8adb3c39e80147091 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:38.824548 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 954/1901] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:38.833695 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7079732-3d27-48ec-99f9-492a63c49174 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:38.834970 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-a7079732-3d27-48ec-99f9-492a63c49174 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:38.919715 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7079732-3d27-48ec-99f9-492a63c49174 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:38.925082 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac84becd-4f32-4611-9339-c25dca3aa753 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:38.926184 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac84becd-4f32-4611-9339-c25dca3aa753 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:38.926720 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac84becd-4f32-4611-9339-c25dca3aa753 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:38.927059 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac84becd-4f32-4611-9339-c25dca3aa753 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:39.012791 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac84becd-4f32-4611-9339-c25dca3aa753 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:39.015333 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac84becd-4f32-4611-9339-c25dca3aa753 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:39.048889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 948/1902] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 216 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:20:39.189599 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6831484-3805-4e30-b0ba-e8ea0deb05d8 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:39.192183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-f6831484-3805-4e30-b0ba-e8ea0deb05d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:39.227513 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6831484-3805-4e30-b0ba-e8ea0deb05d8 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:39.238985 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e987c54-79a5-4372-bd4d-c3f66d83e108 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:39.239775 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e987c54-79a5-4372-bd4d-c3f66d83e108 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:39.240114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e987c54-79a5-4372-bd4d-c3f66d83e108 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:39.240465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e987c54-79a5-4372-bd4d-c3f66d83e108 admin admin] PATH_INFO: `/v3/projects/ea4892e5b25b4462ad734616243f81fa` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:39.242677 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e987c54-79a5-4372-bd4d-c3f66d83e108 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ea4892e5b25b4462ad734616243f81fa)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:39.244700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e987c54-79a5-4372-bd4d-c3f66d83e108 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:39.301853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-3e987c54-79a5-4372-bd4d-c3f66d83e108 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ea4892e5b25b4462ad734616243f81fa', 'request_id': 'req-3e987c54-79a5-4372-bd4d-c3f66d83e108'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:39.308555 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-3e987c54-79a5-4372-bd4d-c3f66d83e108 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ea4892e5b25b4462ad734616243f81fa', 'request_id': 'req-3e987c54-79a5-4372-bd4d-c3f66d83e108'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:39.315821 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-3e987c54-79a5-4372-bd4d-c3f66d83e108 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ea4892e5b25b4462ad734616243f81fa', 'request_id': 'req-3e987c54-79a5-4372-bd4d-c3f66d83e108'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:39.333457 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-3e987c54-79a5-4372-bd4d-c3f66d83e108 admin admin] The token cache is being invalidate because project ea4892e5b25b4462ad734616243f81fa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:39.334464 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-3e987c54-79a5-4372-bd4d-c3f66d83e108 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:39.334889 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-3e987c54-79a5-4372-bd4d-c3f66d83e108 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:39.336420 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 955/1903] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:39 2022] DELETE /identity/v3/projects/ea4892e5b25b4462ad734616243f81fa => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:39.350530 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aacddb3-90b4-462d-bd6d-2990fa402ca5 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:39.350816 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aacddb3-90b4-462d-bd6d-2990fa402ca5 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:39.351089 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aacddb3-90b4-462d-bd6d-2990fa402ca5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:39.370704 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-8aacddb3-90b4-462d-bd6d-2990fa402ca5 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:39.371760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-8aacddb3-90b4-462d-bd6d-2990fa402ca5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:39.389089 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:39.431127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e987c54-79a5-4372-bd4d-c3f66d83e108 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:39.432705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-3e987c54-79a5-4372-bd4d-c3f66d83e108 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:39.448864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 949/1904] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:39.457483 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7feda3da-484e-4ce2-85a6-7510cfeb9e1e None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:39.457822 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7feda3da-484e-4ce2-85a6-7510cfeb9e1e None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:39.462234 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7feda3da-484e-4ce2-85a6-7510cfeb9e1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:39.487203 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-7feda3da-484e-4ce2-85a6-7510cfeb9e1e None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:39.490770 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-7feda3da-484e-4ce2-85a6-7510cfeb9e1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:39.500300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 950/1905] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:39.504513 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7feda3da-484e-4ce2-85a6-7510cfeb9e1e None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:39.514526 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7feda3da-484e-4ce2-85a6-7510cfeb9e1e None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:39.519323 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db43d4af-65cd-475f-b01f-403bee2b4579 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:39.519926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db43d4af-65cd-475f-b01f-403bee2b4579 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:39.520206 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db43d4af-65cd-475f-b01f-403bee2b4579 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:39.520476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db43d4af-65cd-475f-b01f-403bee2b4579 admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:39.521148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db43d4af-65cd-475f-b01f-403bee2b4579 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:20:39.521447 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db43d4af-65cd-475f-b01f-403bee2b4579 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:39.523066 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db43d4af-65cd-475f-b01f-403bee2b4579 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:39.538414 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 951/1906] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:20:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:20:39.538591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e987c54-79a5-4372-bd4d-c3f66d83e108 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:39.540222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db43d4af-65cd-475f-b01f-403bee2b4579 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:39.542851 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98e8af0-6437-45c2-a248-b8c23fa5b66b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:39.543495 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98e8af0-6437-45c2-a248-b8c23fa5b66b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:39.543772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98e8af0-6437-45c2-a248-b8c23fa5b66b admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:39.544045 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98e8af0-6437-45c2-a248-b8c23fa5b66b admin admin] PATH_INFO: `/v3/projects/c5b4f63d1eb24b8eae0fe749cea44218` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:39.545149 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98e8af0-6437-45c2-a248-b8c23fa5b66b admin admin] RBAC: Authorizing `identity:delete_project(project_id=c5b4f63d1eb24b8eae0fe749cea44218)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:39.548378 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98e8af0-6437-45c2-a248-b8c23fa5b66b admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:39.551895 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db43d4af-65cd-475f-b01f-403bee2b4579 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:39.557626 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59dc250-c687-48b0-a854-7a200d69bf06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:39.558366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59dc250-c687-48b0-a854-7a200d69bf06 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:39.558565 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59dc250-c687-48b0-a854-7a200d69bf06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:39.558836 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59dc250-c687-48b0-a854-7a200d69bf06 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:39.559469 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59dc250-c687-48b0-a854-7a200d69bf06 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:39.561367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59dc250-c687-48b0-a854-7a200d69bf06 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:39.589519 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 952/1907] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:39 2022] POST /identity/v3/projects => generated 383 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:20:39.593395 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:39.593969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59dc250-c687-48b0-a854-7a200d69bf06 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:39.603929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a98e8af0-6437-45c2-a248-b8c23fa5b66b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c5b4f63d1eb24b8eae0fe749cea44218', 'request_id': 'req-a98e8af0-6437-45c2-a248-b8c23fa5b66b'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:39.613021 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a98e8af0-6437-45c2-a248-b8c23fa5b66b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c5b4f63d1eb24b8eae0fe749cea44218', 'request_id': 'req-a98e8af0-6437-45c2-a248-b8c23fa5b66b'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:39.620213 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a98e8af0-6437-45c2-a248-b8c23fa5b66b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c5b4f63d1eb24b8eae0fe749cea44218', 'request_id': 'req-a98e8af0-6437-45c2-a248-b8c23fa5b66b'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:39.634951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a98e8af0-6437-45c2-a248-b8c23fa5b66b admin admin] The token cache is being invalidate because project c5b4f63d1eb24b8eae0fe749cea44218 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:39.635240 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a98e8af0-6437-45c2-a248-b8c23fa5b66b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:39.635944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a98e8af0-6437-45c2-a248-b8c23fa5b66b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:39.637473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 956/1908] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:39 2022] DELETE /identity/v3/projects/c5b4f63d1eb24b8eae0fe749cea44218 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:39.641236 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98e8af0-6437-45c2-a248-b8c23fa5b66b admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:39.642537 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a98e8af0-6437-45c2-a248-b8c23fa5b66b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:39.672079 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59dc250-c687-48b0-a854-7a200d69bf06 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:39.676804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38f1438-691c-4175-8871-8af7637ab2f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:39.677409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38f1438-691c-4175-8871-8af7637ab2f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:39.677693 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38f1438-691c-4175-8871-8af7637ab2f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:39.677995 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38f1438-691c-4175-8871-8af7637ab2f1 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:39.679185 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38f1438-691c-4175-8871-8af7637ab2f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:39.680549 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38f1438-691c-4175-8871-8af7637ab2f1 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:39.701385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 953/1909] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:39 2022] POST /identity/v3/users => generated 292 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:39.705193 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38f1438-691c-4175-8871-8af7637ab2f1 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:39.706846 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-a38f1438-691c-4175-8871-8af7637ab2f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:39.711524 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98e8af0-6437-45c2-a248-b8c23fa5b66b admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:39.716170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8770429d-7253-4ee3-a131-6b478959d3b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:39.716797 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8770429d-7253-4ee3-a131-6b478959d3b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:39.717065 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8770429d-7253-4ee3-a131-6b478959d3b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:39.717337 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8770429d-7253-4ee3-a131-6b478959d3b1 admin admin] PATH_INFO: `/v3/users/a853049913cb45f2b473c266d12f73ae` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:39.718903 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38f1438-691c-4175-8871-8af7637ab2f1 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:39.719012 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8770429d-7253-4ee3-a131-6b478959d3b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a853049913cb45f2b473c266d12f73ae)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:39.720890 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8770429d-7253-4ee3-a131-6b478959d3b1 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:39.723596 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31db3383-0459-4a69-937b-170775746594 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:39.724189 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31db3383-0459-4a69-937b-170775746594 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:39.724464 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31db3383-0459-4a69-937b-170775746594 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:39.724733 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31db3383-0459-4a69-937b-170775746594 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:39.725364 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31db3383-0459-4a69-937b-170775746594 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:39.726961 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31db3383-0459-4a69-937b-170775746594 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:39.741776 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 954/1910] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:39 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:20:39.743433 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31db3383-0459-4a69-937b-170775746594 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:39.753604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31db3383-0459-4a69-937b-170775746594 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:39.758967 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b4df59-379a-4f13-a2b4-a9c42b7b4d02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:39.759586 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b4df59-379a-4f13-a2b4-a9c42b7b4d02 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:39.759865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b4df59-379a-4f13-a2b4-a9c42b7b4d02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:39.760136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b4df59-379a-4f13-a2b4-a9c42b7b4d02 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:39.760946 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81b4df59-379a-4f13-a2b4-a9c42b7b4d02 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:39.762408 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81b4df59-379a-4f13-a2b4-a9c42b7b4d02 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:39.774258 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8770429d-7253-4ee3-a131-6b478959d3b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a853049913cb45f2b473c266d12f73ae'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:39.777984 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 955/1911] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:39 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:20:39.779737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b4df59-379a-4f13-a2b4-a9c42b7b4d02 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:39.784000 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8770429d-7253-4ee3-a131-6b478959d3b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a853049913cb45f2b473c266d12f73ae'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:39.785403 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8770429d-7253-4ee3-a131-6b478959d3b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a853049913cb45f2b473c266d12f73ae'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:39.793310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b4df59-379a-4f13-a2b4-a9c42b7b4d02 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:39.795365 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8770429d-7253-4ee3-a131-6b478959d3b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a853049913cb45f2b473c266d12f73ae'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:39.799066 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af79601-7868-4f7f-9a4a-611a72229b5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:39.799732 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af79601-7868-4f7f-9a4a-611a72229b5d admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:39.800040 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af79601-7868-4f7f-9a4a-611a72229b5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:39.800322 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af79601-7868-4f7f-9a4a-611a72229b5d admin admin] PATH_INFO: `/v3/projects/cfe73812e575488ab8984bce1a0246ad/users/aef9d5924eb44c74ae86013a6d3e6318/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:39.812463 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8770429d-7253-4ee3-a131-6b478959d3b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a853049913cb45f2b473c266d12f73ae'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:39.815877 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af79601-7868-4f7f-9a4a-611a72229b5d admin admin] RBAC: Authorizing `identity:create_grant(project_id=cfe73812e575488ab8984bce1a0246ad, user_id=aef9d5924eb44c74ae86013a6d3e6318, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:39.818464 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af79601-7868-4f7f-9a4a-611a72229b5d admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:39.820464 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 957/1912] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:39 2022] DELETE /identity/v3/users/a853049913cb45f2b473c266d12f73ae => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:20:39.833831 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 956/1913] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:39 2022] PUT /identity/v3/projects/cfe73812e575488ab8984bce1a0246ad/users/aef9d5924eb44c74ae86013a6d3e6318/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:39.844693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09950dc-1b07-4598-aca7-f63f7461edc5 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:39.844974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09950dc-1b07-4598-aca7-f63f7461edc5 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:39.845243 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09950dc-1b07-4598-aca7-f63f7461edc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:39.873695 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-b09950dc-1b07-4598-aca7-f63f7461edc5 None None] MFA Rules not processed for user `aef9d5924eb44c74ae86013a6d3e6318`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:39.874955 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-b09950dc-1b07-4598-aca7-f63f7461edc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:39.909818 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:39.943830 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 958/1914] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:39 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:39.948637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af79601-7868-4f7f-9a4a-611a72229b5d admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:39.950392 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-5af79601-7868-4f7f-9a4a-611a72229b5d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:39.982550 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:40.027676 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af79601-7868-4f7f-9a4a-611a72229b5d admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:40.033178 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7520c36c-0efc-4341-8a7e-9efa428aac89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:40.033819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7520c36c-0efc-4341-8a7e-9efa428aac89 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:40.034097 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7520c36c-0efc-4341-8a7e-9efa428aac89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:40.034366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7520c36c-0efc-4341-8a7e-9efa428aac89 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:40.034999 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7520c36c-0efc-4341-8a7e-9efa428aac89 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:40.036865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7520c36c-0efc-4341-8a7e-9efa428aac89 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:40.070236 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 957/1915] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:39 2022] POST /identity/v3/projects => generated 381 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:40.074377 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09950dc-1b07-4598-aca7-f63f7461edc5 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:40.086050 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09950dc-1b07-4598-aca7-f63f7461edc5 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:40.148915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c431804f-2d93-47a7-9ead-16ab0ee746f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:40.149489 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c431804f-2d93-47a7-9ead-16ab0ee746f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:40.150179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c431804f-2d93-47a7-9ead-16ab0ee746f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:40.150179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c431804f-2d93-47a7-9ead-16ab0ee746f9 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:40.150712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c431804f-2d93-47a7-9ead-16ab0ee746f9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:40.152630 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c431804f-2d93-47a7-9ead-16ab0ee746f9 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:40.193442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7520c36c-0efc-4341-8a7e-9efa428aac89 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:40.195164 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-7520c36c-0efc-4341-8a7e-9efa428aac89 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:40.200519 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 959/1916] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:40 2022] POST /identity/v3/users => generated 297 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:20:40.201696 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c431804f-2d93-47a7-9ead-16ab0ee746f9 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:40.213934 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7520c36c-0efc-4341-8a7e-9efa428aac89 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:40.214536 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c431804f-2d93-47a7-9ead-16ab0ee746f9 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:40.214760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f482296-a243-428b-9a92-c002aff81982 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:40.215533 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f482296-a243-428b-9a92-c002aff81982 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:40.215772 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f482296-a243-428b-9a92-c002aff81982 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:40.216044 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f482296-a243-428b-9a92-c002aff81982 admin admin] PATH_INFO: `/v3/projects/3a7201c9aa744077b931b70cdbc7295b` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:40.217140 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f482296-a243-428b-9a92-c002aff81982 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3a7201c9aa744077b931b70cdbc7295b)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:40.219803 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f482296-a243-428b-9a92-c002aff81982 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:40.219973 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122d7f4c-1640-4b95-a079-ab5e4557442c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:40.220564 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122d7f4c-1640-4b95-a079-ab5e4557442c admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:40.220845 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122d7f4c-1640-4b95-a079-ab5e4557442c admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:40.221120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122d7f4c-1640-4b95-a079-ab5e4557442c admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:40.221801 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122d7f4c-1640-4b95-a079-ab5e4557442c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:40.223397 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122d7f4c-1640-4b95-a079-ab5e4557442c admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:40.239075 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 960/1917] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:40 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:40.243412 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122d7f4c-1640-4b95-a079-ab5e4557442c admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:40.258292 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122d7f4c-1640-4b95-a079-ab5e4557442c admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:40.265572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c50359c-56bb-4dc7-9fa7-27860cc4b89b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:40.266663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c50359c-56bb-4dc7-9fa7-27860cc4b89b admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:40.266920 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c50359c-56bb-4dc7-9fa7-27860cc4b89b admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:40.267207 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c50359c-56bb-4dc7-9fa7-27860cc4b89b admin admin] PATH_INFO: `/v3/projects/37249b979d864db6a2af88e936781380/users/d1edc763424a40cd962f9cd517e1ffee/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:40.286540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c50359c-56bb-4dc7-9fa7-27860cc4b89b admin admin] RBAC: Authorizing `identity:create_grant(project_id=37249b979d864db6a2af88e936781380, user_id=d1edc763424a40cd962f9cd517e1ffee, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:40.289510 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c50359c-56bb-4dc7-9fa7-27860cc4b89b admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:40.312618 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-9f482296-a243-428b-9a92-c002aff81982 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3a7201c9aa744077b931b70cdbc7295b', 'request_id': 'req-9f482296-a243-428b-9a92-c002aff81982'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:40.319770 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-9f482296-a243-428b-9a92-c002aff81982 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3a7201c9aa744077b931b70cdbc7295b', 'request_id': 'req-9f482296-a243-428b-9a92-c002aff81982'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:40.321588 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 961/1918] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:40 2022] PUT /identity/v3/projects/37249b979d864db6a2af88e936781380/users/d1edc763424a40cd962f9cd517e1ffee/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:20:40.325950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c50359c-56bb-4dc7-9fa7-27860cc4b89b admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:40.328109 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-9f482296-a243-428b-9a92-c002aff81982 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3a7201c9aa744077b931b70cdbc7295b', 'request_id': 'req-9f482296-a243-428b-9a92-c002aff81982'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:40.345886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-9f482296-a243-428b-9a92-c002aff81982 admin admin] The token cache is being invalidate because project 3a7201c9aa744077b931b70cdbc7295b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:40.346622 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-9f482296-a243-428b-9a92-c002aff81982 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:40.347274 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-9f482296-a243-428b-9a92-c002aff81982 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:40.351037 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 958/1919] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:40 2022] DELETE /identity/v3/projects/3a7201c9aa744077b931b70cdbc7295b => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:20:40.356586 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097dc2b3-466b-4cf7-aab2-8ea9cd8e5921 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:40.356586 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097dc2b3-466b-4cf7-aab2-8ea9cd8e5921 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:40.356954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097dc2b3-466b-4cf7-aab2-8ea9cd8e5921 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:40.387751 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-097dc2b3-466b-4cf7-aab2-8ea9cd8e5921 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:40.389253 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-097dc2b3-466b-4cf7-aab2-8ea9cd8e5921 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:40.422335 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c50359c-56bb-4dc7-9fa7-27860cc4b89b admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:40.430387 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23baa6ce-c55a-4aa5-afcf-cc5cae989120 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:40.430946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23baa6ce-c55a-4aa5-afcf-cc5cae989120 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:40.431231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23baa6ce-c55a-4aa5-afcf-cc5cae989120 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:40.431632 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23baa6ce-c55a-4aa5-afcf-cc5cae989120 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:40.432292 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23baa6ce-c55a-4aa5-afcf-cc5cae989120 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:40.435800 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23baa6ce-c55a-4aa5-afcf-cc5cae989120 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:40.448120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 962/1920] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:40 2022] GET /identity/v3/roles => generated 1420 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:40.452922 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23baa6ce-c55a-4aa5-afcf-cc5cae989120 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:40.454755 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-23baa6ce-c55a-4aa5-afcf-cc5cae989120 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:40.468886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 959/1921] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:40.470422 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23baa6ce-c55a-4aa5-afcf-cc5cae989120 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:40.470595 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61093aa1-1534-460a-b43e-f0f54ff23e7a None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:40.470944 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61093aa1-1534-460a-b43e-f0f54ff23e7a None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:40.471220 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61093aa1-1534-460a-b43e-f0f54ff23e7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:40.482444 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29298805-1ea3-4185-abe3-35bd11e205f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:40.483575 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29298805-1ea3-4185-abe3-35bd11e205f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:40.483919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29298805-1ea3-4185-abe3-35bd11e205f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:40.484335 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29298805-1ea3-4185-abe3-35bd11e205f7 admin admin] PATH_INFO: `/v3/projects/37249b979d864db6a2af88e936781380/users/d1edc763424a40cd962f9cd517e1ffee/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:40.489689 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29298805-1ea3-4185-abe3-35bd11e205f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=37249b979d864db6a2af88e936781380, user_id=d1edc763424a40cd962f9cd517e1ffee, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:40.492439 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29298805-1ea3-4185-abe3-35bd11e205f7 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:40.494349 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-61093aa1-1534-460a-b43e-f0f54ff23e7a None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:40.496688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-61093aa1-1534-460a-b43e-f0f54ff23e7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:40.517800 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 963/1922] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:40 2022] PUT /identity/v3/projects/37249b979d864db6a2af88e936781380/users/d1edc763424a40cd962f9cd517e1ffee/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:40.520010 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd834ed-e8c6-401f-a673-2036d4ef5d46 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:40.520417 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd834ed-e8c6-401f-a673-2036d4ef5d46 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:40.520690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd834ed-e8c6-401f-a673-2036d4ef5d46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:40.534461 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:40.566359 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:40.574729 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 960/1923] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:40.578854 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-4dd834ed-e8c6-401f-a673-2036d4ef5d46 None None] MFA Rules not processed for user `d1edc763424a40cd962f9cd517e1ffee`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:40.579617 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61093aa1-1534-460a-b43e-f0f54ff23e7a None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:40.580281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-4dd834ed-e8c6-401f-a673-2036d4ef5d46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:40.588985 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61093aa1-1534-460a-b43e-f0f54ff23e7a None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:40.593903 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018b9701-1933-4621-8136-1cb3678812bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:40.594502 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018b9701-1933-4621-8136-1cb3678812bc admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:40.594784 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018b9701-1933-4621-8136-1cb3678812bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:40.595133 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018b9701-1933-4621-8136-1cb3678812bc admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:40.595878 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018b9701-1933-4621-8136-1cb3678812bc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:20:40.596181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018b9701-1933-4621-8136-1cb3678812bc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:40.598378 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018b9701-1933-4621-8136-1cb3678812bc admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:40.617824 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 961/1924] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:20:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:20:40.618771 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018b9701-1933-4621-8136-1cb3678812bc admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:40.632446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018b9701-1933-4621-8136-1cb3678812bc admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:40.638780 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9a2fc7-268c-486c-bd28-bec3f0bcd3d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:40.640253 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9a2fc7-268c-486c-bd28-bec3f0bcd3d3 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:40.640793 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9a2fc7-268c-486c-bd28-bec3f0bcd3d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:40.641081 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9a2fc7-268c-486c-bd28-bec3f0bcd3d3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:40.643873 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9a2fc7-268c-486c-bd28-bec3f0bcd3d3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:40.643988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9a2fc7-268c-486c-bd28-bec3f0bcd3d3 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:40.683759 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 964/1925] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:40 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:40.685772 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 962/1926] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:40 2022] POST /identity/v3/projects => generated 373 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:20:40.691753 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd834ed-e8c6-401f-a673-2036d4ef5d46 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:40.698306 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9a2fc7-268c-486c-bd28-bec3f0bcd3d3 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:40.701056 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-5e9a2fc7-268c-486c-bd28-bec3f0bcd3d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:40.710286 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd834ed-e8c6-401f-a673-2036d4ef5d46 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:40.776365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9a2fc7-268c-486c-bd28-bec3f0bcd3d3 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:40.781457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f98748d-a73f-431f-b53d-1f9b4da9052e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:40.782257 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f98748d-a73f-431f-b53d-1f9b4da9052e service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:40.783188 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f98748d-a73f-431f-b53d-1f9b4da9052e service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:40.783741 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f98748d-a73f-431f-b53d-1f9b4da9052e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:40.799035 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8465cf8f-6dab-4319-b045-ec100b8f4da2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:40.799651 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8465cf8f-6dab-4319-b045-ec100b8f4da2 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:40.799927 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8465cf8f-6dab-4319-b045-ec100b8f4da2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:40.800197 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8465cf8f-6dab-4319-b045-ec100b8f4da2 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:40.801196 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8465cf8f-6dab-4319-b045-ec100b8f4da2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:40.803891 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8465cf8f-6dab-4319-b045-ec100b8f4da2 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:40.830277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 965/1927] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:40 2022] POST /identity/v3/users => generated 287 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:40.835217 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8465cf8f-6dab-4319-b045-ec100b8f4da2 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:40.846320 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8465cf8f-6dab-4319-b045-ec100b8f4da2 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:40.851343 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6454c09-9383-4f23-b579-83305d4f369b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:40.852096 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6454c09-9383-4f23-b579-83305d4f369b admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:40.852381 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6454c09-9383-4f23-b579-83305d4f369b admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:40.852650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6454c09-9383-4f23-b579-83305d4f369b admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:40.853292 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6454c09-9383-4f23-b579-83305d4f369b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:40.855026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6454c09-9383-4f23-b579-83305d4f369b admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:40.869794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 966/1928] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:40 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:20:40.871660 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6454c09-9383-4f23-b579-83305d4f369b admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:40.882508 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6454c09-9383-4f23-b579-83305d4f369b admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:40.885567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f98748d-a73f-431f-b53d-1f9b4da9052e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:40.887040 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f98748d-a73f-431f-b53d-1f9b4da9052e service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:40.887354 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc2b533-34b0-45c6-8b36-2fdf3822cbc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:40.887953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc2b533-34b0-45c6-8b36-2fdf3822cbc3 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:40.888231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc2b533-34b0-45c6-8b36-2fdf3822cbc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:40.888504 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc2b533-34b0-45c6-8b36-2fdf3822cbc3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:40.889136 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc2b533-34b0-45c6-8b36-2fdf3822cbc3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:40.890793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc2b533-34b0-45c6-8b36-2fdf3822cbc3 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:40.896633 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 963/1929] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:40 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 199 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:20:40.897423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f98748d-a73f-431f-b53d-1f9b4da9052e service cinder] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:40.898723 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-7f98748d-a73f-431f-b53d-1f9b4da9052e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:40.905785 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 967/1930] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:40 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:40.907821 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc2b533-34b0-45c6-8b36-2fdf3822cbc3 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:40.912356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f98748d-a73f-431f-b53d-1f9b4da9052e service cinder] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:40.917939 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6e247a-05ff-46f5-930e-8a4aae8088e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:40.918571 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6e247a-05ff-46f5-930e-8a4aae8088e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:40.918851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6e247a-05ff-46f5-930e-8a4aae8088e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:40.919127 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6e247a-05ff-46f5-930e-8a4aae8088e1 admin admin] PATH_INFO: `/v3/users/fea4a3a9230f416082f438b9f9e481ad` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:40.919535 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc2b533-34b0-45c6-8b36-2fdf3822cbc3 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:40.920284 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6e247a-05ff-46f5-930e-8a4aae8088e1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fea4a3a9230f416082f438b9f9e481ad)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:40.922859 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6e247a-05ff-46f5-930e-8a4aae8088e1 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:40.925324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997cdfeb-b2ff-40ed-b2fc-ae189ab47103 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:40.926145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997cdfeb-b2ff-40ed-b2fc-ae189ab47103 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:40.926424 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997cdfeb-b2ff-40ed-b2fc-ae189ab47103 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:40.926821 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997cdfeb-b2ff-40ed-b2fc-ae189ab47103 admin admin] PATH_INFO: `/v3/projects/1ffa7078248541158035d49ab4254b73/users/d4abb7018fa64528ad601dadb8123aa1/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:40.942440 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997cdfeb-b2ff-40ed-b2fc-ae189ab47103 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ffa7078248541158035d49ab4254b73, user_id=d4abb7018fa64528ad601dadb8123aa1, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:40.945148 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997cdfeb-b2ff-40ed-b2fc-ae189ab47103 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:40.962230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 968/1931] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:40 2022] PUT /identity/v3/projects/1ffa7078248541158035d49ab4254b73/users/d4abb7018fa64528ad601dadb8123aa1/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:20:40.967468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b76961-8426-45df-b6b0-1989e0931b0b None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:40.967793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b76961-8426-45df-b6b0-1989e0931b0b None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:40.968907 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b76961-8426-45df-b6b0-1989e0931b0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:40.977088 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=93,keystone:INSERT=1,keystone:DELETE=6 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:40.980317 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8f6e247a-05ff-46f5-930e-8a4aae8088e1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fea4a3a9230f416082f438b9f9e481ad'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:40.991333 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8f6e247a-05ff-46f5-930e-8a4aae8088e1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fea4a3a9230f416082f438b9f9e481ad'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:40.991823 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-d4b76961-8426-45df-b6b0-1989e0931b0b None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:40.992181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8f6e247a-05ff-46f5-930e-8a4aae8088e1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fea4a3a9230f416082f438b9f9e481ad'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:40.993065 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d4b76961-8426-45df-b6b0-1989e0931b0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:41.001935 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8f6e247a-05ff-46f5-930e-8a4aae8088e1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fea4a3a9230f416082f438b9f9e481ad'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:41.019334 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8f6e247a-05ff-46f5-930e-8a4aae8088e1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fea4a3a9230f416082f438b9f9e481ad'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:41.027077 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 964/1932] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:40 2022] DELETE /identity/v3/users/fea4a3a9230f416082f438b9f9e481ad => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:41.028348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6e247a-05ff-46f5-930e-8a4aae8088e1 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:41.030717 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-8f6e247a-05ff-46f5-930e-8a4aae8088e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:41.038680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:41.063231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 969/1933] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:20:41.064762 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96dae052-e5ef-4c73-95d9-05b9a261adca None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:41.065162 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96dae052-e5ef-4c73-95d9-05b9a261adca None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:41.065507 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96dae052-e5ef-4c73-95d9-05b9a261adca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:41.086856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-96dae052-e5ef-4c73-95d9-05b9a261adca None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:41.088159 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-96dae052-e5ef-4c73-95d9-05b9a261adca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:41.103163 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6e247a-05ff-46f5-930e-8a4aae8088e1 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:41.107744 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090daa1a-cc77-4742-85d1-1eb34faebc6b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:41.108969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090daa1a-cc77-4742-85d1-1eb34faebc6b service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:41.109380 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090daa1a-cc77-4742-85d1-1eb34faebc6b service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:41.109657 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090daa1a-cc77-4742-85d1-1eb34faebc6b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:41.111487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-090daa1a-cc77-4742-85d1-1eb34faebc6b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:41.154181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 970/1934] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:41.155948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96dae052-e5ef-4c73-95d9-05b9a261adca None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:41.157384 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-96dae052-e5ef-4c73-95d9-05b9a261adca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:41.188915 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090daa1a-cc77-4742-85d1-1eb34faebc6b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:41.190531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090daa1a-cc77-4742-85d1-1eb34faebc6b service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:41.200335 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 965/1935] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:41 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:41.201205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090daa1a-cc77-4742-85d1-1eb34faebc6b service cinder] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:41.202911 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-090daa1a-cc77-4742-85d1-1eb34faebc6b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:41.227367 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96dae052-e5ef-4c73-95d9-05b9a261adca None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:41.231352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373a4472-f370-432d-a3d5-9a50a1112339 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:41.232105 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373a4472-f370-432d-a3d5-9a50a1112339 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:41.232463 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373a4472-f370-432d-a3d5-9a50a1112339 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:41.232739 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373a4472-f370-432d-a3d5-9a50a1112339 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:41.243928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373a4472-f370-432d-a3d5-9a50a1112339 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:41.245366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373a4472-f370-432d-a3d5-9a50a1112339 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:41.256770 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 971/1936] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:41.276288 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090daa1a-cc77-4742-85d1-1eb34faebc6b service cinder] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:41.284400 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c29b2a-b905-41e3-a551-68ba8e7dc9c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:41.285065 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c29b2a-b905-41e3-a551-68ba8e7dc9c7 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:41.285348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c29b2a-b905-41e3-a551-68ba8e7dc9c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:41.285644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c29b2a-b905-41e3-a551-68ba8e7dc9c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:41.297681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c29b2a-b905-41e3-a551-68ba8e7dc9c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:41.299342 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c29b2a-b905-41e3-a551-68ba8e7dc9c7 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:41.311895 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 966/1937] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:41.594648 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373a4472-f370-432d-a3d5-9a50a1112339 service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:41.597088 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-373a4472-f370-432d-a3d5-9a50a1112339 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:41.622998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373a4472-f370-432d-a3d5-9a50a1112339 service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:41.638985 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6ec6ef-8ce1-4fab-8cfe-391f57871cf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:41.641167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6ec6ef-8ce1-4fab-8cfe-391f57871cf0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:41.641562 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6ec6ef-8ce1-4fab-8cfe-391f57871cf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:41.641928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6ec6ef-8ce1-4fab-8cfe-391f57871cf0 admin admin] PATH_INFO: `/v3/users/2ca715dd3543456daa09c9bbc9d21349` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:41.646300 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6ec6ef-8ce1-4fab-8cfe-391f57871cf0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2ca715dd3543456daa09c9bbc9d21349)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:41.648450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6ec6ef-8ce1-4fab-8cfe-391f57871cf0 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:41.778423 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6c6ec6ef-8ce1-4fab-8cfe-391f57871cf0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2ca715dd3543456daa09c9bbc9d21349'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:41.790401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6c6ec6ef-8ce1-4fab-8cfe-391f57871cf0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2ca715dd3543456daa09c9bbc9d21349'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:41.791300 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6c6ec6ef-8ce1-4fab-8cfe-391f57871cf0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2ca715dd3543456daa09c9bbc9d21349'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:41.801239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6c6ec6ef-8ce1-4fab-8cfe-391f57871cf0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2ca715dd3543456daa09c9bbc9d21349'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:41.834349 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6c6ec6ef-8ce1-4fab-8cfe-391f57871cf0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2ca715dd3543456daa09c9bbc9d21349'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:41.838823 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:41.849318 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 972/1938] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:41 2022] DELETE /identity/v3/users/2ca715dd3543456daa09c9bbc9d21349 => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:41.854110 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db3c50c-e6a9-4616-b0b5-35c2755a17b8 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:41.854603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db3c50c-e6a9-4616-b0b5-35c2755a17b8 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:41.854603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db3c50c-e6a9-4616-b0b5-35c2755a17b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:41.893000 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-3db3c50c-e6a9-4616-b0b5-35c2755a17b8 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:41.896800 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-3db3c50c-e6a9-4616-b0b5-35c2755a17b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:41.930593 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:DELETE=4,keystone:SELECT=95,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:41.987259 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6ec6ef-8ce1-4fab-8cfe-391f57871cf0 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:41.989868 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6c6ec6ef-8ce1-4fab-8cfe-391f57871cf0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:42.058265 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 967/1939] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:42.073362 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db3c50c-e6a9-4616-b0b5-35c2755a17b8 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:42.075197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-3db3c50c-e6a9-4616-b0b5-35c2755a17b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:42.170150 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6ec6ef-8ce1-4fab-8cfe-391f57871cf0 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:42.188577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b628dc-9e40-48d8-b853-098d13e5312f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:42.189669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b628dc-9e40-48d8-b853-098d13e5312f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:42.190061 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b628dc-9e40-48d8-b853-098d13e5312f admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:42.190411 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b628dc-9e40-48d8-b853-098d13e5312f admin admin] PATH_INFO: `/v3/projects/23cf7efa73b14cab92137fda57640001` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:42.195517 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b628dc-9e40-48d8-b853-098d13e5312f admin admin] RBAC: Authorizing `identity:delete_project(project_id=23cf7efa73b14cab92137fda57640001)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:42.197584 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b628dc-9e40-48d8-b853-098d13e5312f admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:42.206466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db3c50c-e6a9-4616-b0b5-35c2755a17b8 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:42.223206 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9750114f-c4c6-491f-a80c-99d5f6ae646e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:42.223806 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9750114f-c4c6-491f-a80c-99d5f6ae646e service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:42.224088 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9750114f-c4c6-491f-a80c-99d5f6ae646e service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:42.224360 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9750114f-c4c6-491f-a80c-99d5f6ae646e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:42.237482 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9750114f-c4c6-491f-a80c-99d5f6ae646e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:42.239053 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9750114f-c4c6-491f-a80c-99d5f6ae646e service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:42.253403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 968/1940] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:42.304610 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-82b628dc-9e40-48d8-b853-098d13e5312f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '23cf7efa73b14cab92137fda57640001', 'request_id': 'req-82b628dc-9e40-48d8-b853-098d13e5312f'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:42.317591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-82b628dc-9e40-48d8-b853-098d13e5312f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '23cf7efa73b14cab92137fda57640001', 'request_id': 'req-82b628dc-9e40-48d8-b853-098d13e5312f'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:42.335422 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-82b628dc-9e40-48d8-b853-098d13e5312f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '23cf7efa73b14cab92137fda57640001', 'request_id': 'req-82b628dc-9e40-48d8-b853-098d13e5312f'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:42.361264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-82b628dc-9e40-48d8-b853-098d13e5312f admin admin] The token cache is being invalidate because project 23cf7efa73b14cab92137fda57640001 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:42.361562 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-82b628dc-9e40-48d8-b853-098d13e5312f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:42.363973 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-82b628dc-9e40-48d8-b853-098d13e5312f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:42.366983 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 973/1941] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:41 2022] DELETE /identity/v3/projects/23cf7efa73b14cab92137fda57640001 => generated 0 bytes in 381 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:42.371102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9750114f-c4c6-491f-a80c-99d5f6ae646e service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:42.384983 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9750114f-c4c6-491f-a80c-99d5f6ae646e service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:42.459652 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d34c54-7b96-49d2-b323-fde63caf7b2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:42.460876 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d34c54-7b96-49d2-b323-fde63caf7b2e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:42.461381 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d34c54-7b96-49d2-b323-fde63caf7b2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:42.461825 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d34c54-7b96-49d2-b323-fde63caf7b2e admin admin] PATH_INFO: `/v3/users/4663f0df277a40f496432e95ea3445c7` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:42.463215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d34c54-7b96-49d2-b323-fde63caf7b2e admin admin] RBAC: Authorizing `identity:delete_user(user_id=4663f0df277a40f496432e95ea3445c7)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:42.465847 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d34c54-7b96-49d2-b323-fde63caf7b2e admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:42.534447 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-25d34c54-7b96-49d2-b323-fde63caf7b2e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4663f0df277a40f496432e95ea3445c7'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:42.544368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-25d34c54-7b96-49d2-b323-fde63caf7b2e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4663f0df277a40f496432e95ea3445c7'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:42.545489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-25d34c54-7b96-49d2-b323-fde63caf7b2e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4663f0df277a40f496432e95ea3445c7'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:42.556852 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-25d34c54-7b96-49d2-b323-fde63caf7b2e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4663f0df277a40f496432e95ea3445c7'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:42.558370 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:42.588363 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-25d34c54-7b96-49d2-b323-fde63caf7b2e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4663f0df277a40f496432e95ea3445c7'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:42.604745 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 969/1942] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:42 2022] DELETE /identity/v3/users/4663f0df277a40f496432e95ea3445c7 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:42.912172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b628dc-9e40-48d8-b853-098d13e5312f admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:42.914093 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-82b628dc-9e40-48d8-b853-098d13e5312f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:42.933777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b628dc-9e40-48d8-b853-098d13e5312f admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:42.989349 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb84b22-61e9-4c39-af43-106eede03d9c None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:42.989947 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb84b22-61e9-4c39-af43-106eede03d9c None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:42.990522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb84b22-61e9-4c39-af43-106eede03d9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:43.055666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-3bb84b22-61e9-4c39-af43-106eede03d9c None None] MFA Rules not processed for user `d4abb7018fa64528ad601dadb8123aa1`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:43.058475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-3bb84b22-61e9-4c39-af43-106eede03d9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:43.090498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121a6673-fc4e-4ff8-a234-34de453eecaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:43.092073 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121a6673-fc4e-4ff8-a234-34de453eecaa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:43.092536 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121a6673-fc4e-4ff8-a234-34de453eecaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:43.093676 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121a6673-fc4e-4ff8-a234-34de453eecaa admin admin] PATH_INFO: `/v3/projects/1ff813b3184047249b2ad5ab9d6743f7` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:43.095869 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121a6673-fc4e-4ff8-a234-34de453eecaa admin admin] RBAC: Authorizing `identity:delete_project(project_id=1ff813b3184047249b2ad5ab9d6743f7)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:43.098063 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121a6673-fc4e-4ff8-a234-34de453eecaa admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:43.183535 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:43.195754 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 970/1943] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:42 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:20:43.209556 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb84b22-61e9-4c39-af43-106eede03d9c None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:43.214357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-3bb84b22-61e9-4c39-af43-106eede03d9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:43.225097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-121a6673-fc4e-4ff8-a234-34de453eecaa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1ff813b3184047249b2ad5ab9d6743f7', 'request_id': 'req-121a6673-fc4e-4ff8-a234-34de453eecaa'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:43.236789 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-121a6673-fc4e-4ff8-a234-34de453eecaa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1ff813b3184047249b2ad5ab9d6743f7', 'request_id': 'req-121a6673-fc4e-4ff8-a234-34de453eecaa'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:43.244258 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb84b22-61e9-4c39-af43-106eede03d9c None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:43.246485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-121a6673-fc4e-4ff8-a234-34de453eecaa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1ff813b3184047249b2ad5ab9d6743f7', 'request_id': 'req-121a6673-fc4e-4ff8-a234-34de453eecaa'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:43.271290 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-121a6673-fc4e-4ff8-a234-34de453eecaa admin admin] The token cache is being invalidate because project 1ff813b3184047249b2ad5ab9d6743f7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:43.271692 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-121a6673-fc4e-4ff8-a234-34de453eecaa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:43.272366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-121a6673-fc4e-4ff8-a234-34de453eecaa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:43.274537 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 974/1944] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:42 2022] DELETE /identity/v3/projects/1ff813b3184047249b2ad5ab9d6743f7 => generated 0 bytes in 363 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:43.276748 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121a6673-fc4e-4ff8-a234-34de453eecaa admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:43.278830 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-121a6673-fc4e-4ff8-a234-34de453eecaa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:43.342587 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3294d1-7243-4db3-8e10-9c75127c0769 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:43.345155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3294d1-7243-4db3-8e10-9c75127c0769 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:43.345469 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3294d1-7243-4db3-8e10-9c75127c0769 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:43.346050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3294d1-7243-4db3-8e10-9c75127c0769 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:43.346733 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3294d1-7243-4db3-8e10-9c75127c0769 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:43.350399 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3294d1-7243-4db3-8e10-9c75127c0769 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:43.358258 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121a6673-fc4e-4ff8-a234-34de453eecaa admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:43.363397 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc415e8-ce0f-4c98-a327-d8a0ebce4dc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:43.364155 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc415e8-ce0f-4c98-a327-d8a0ebce4dc8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:43.364445 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc415e8-ce0f-4c98-a327-d8a0ebce4dc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:43.364928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc415e8-ce0f-4c98-a327-d8a0ebce4dc8 admin admin] PATH_INFO: `/v3/projects/a25926354d504c84a348bbe506b83b3a` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:43.366135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc415e8-ce0f-4c98-a327-d8a0ebce4dc8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a25926354d504c84a348bbe506b83b3a)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:43.368221 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc415e8-ce0f-4c98-a327-d8a0ebce4dc8 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:43.399568 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 971/1945] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:43 2022] POST /identity/v3/projects => generated 373 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:43.401835 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3294d1-7243-4db3-8e10-9c75127c0769 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:43.404225 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-2a3294d1-7243-4db3-8e10-9c75127c0769 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:43.448425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-3fc415e8-ce0f-4c98-a327-d8a0ebce4dc8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a25926354d504c84a348bbe506b83b3a', 'request_id': 'req-3fc415e8-ce0f-4c98-a327-d8a0ebce4dc8'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:43.455494 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-3fc415e8-ce0f-4c98-a327-d8a0ebce4dc8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a25926354d504c84a348bbe506b83b3a', 'request_id': 'req-3fc415e8-ce0f-4c98-a327-d8a0ebce4dc8'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:43.467249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-3fc415e8-ce0f-4c98-a327-d8a0ebce4dc8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a25926354d504c84a348bbe506b83b3a', 'request_id': 'req-3fc415e8-ce0f-4c98-a327-d8a0ebce4dc8'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:43.490106 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-3fc415e8-ce0f-4c98-a327-d8a0ebce4dc8 admin admin] The token cache is being invalidate because project a25926354d504c84a348bbe506b83b3a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:43.490639 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-3fc415e8-ce0f-4c98-a327-d8a0ebce4dc8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:43.492121 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-3fc415e8-ce0f-4c98-a327-d8a0ebce4dc8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:43.495615 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:DELETE=2,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:43.496198 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 975/1946] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:43 2022] DELETE /identity/v3/projects/a25926354d504c84a348bbe506b83b3a => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:43.496967 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc415e8-ce0f-4c98-a327-d8a0ebce4dc8 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:43.499535 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-3fc415e8-ce0f-4c98-a327-d8a0ebce4dc8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:43.513673 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3294d1-7243-4db3-8e10-9c75127c0769 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:43.556908 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:43.579153 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc415e8-ce0f-4c98-a327-d8a0ebce4dc8 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:43.579619 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2a78b0-a761-4429-8940-bbd8ee547f05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:43.580377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2a78b0-a761-4429-8940-bbd8ee547f05 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:43.580664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2a78b0-a761-4429-8940-bbd8ee547f05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:43.581220 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2a78b0-a761-4429-8940-bbd8ee547f05 admin admin] PATH_INFO: `/v3/users/a4b986d177014d749891075e7692d902` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:43.582381 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2a78b0-a761-4429-8940-bbd8ee547f05 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a4b986d177014d749891075e7692d902)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:43.585271 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e28d7f4-b5b7-4602-a0da-bf3dcf6ac5d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:43.585532 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2a78b0-a761-4429-8940-bbd8ee547f05 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:43.585923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e28d7f4-b5b7-4602-a0da-bf3dcf6ac5d0 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:43.586289 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e28d7f4-b5b7-4602-a0da-bf3dcf6ac5d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:43.586567 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e28d7f4-b5b7-4602-a0da-bf3dcf6ac5d0 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:43.600584 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e28d7f4-b5b7-4602-a0da-bf3dcf6ac5d0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:43.602760 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e28d7f4-b5b7-4602-a0da-bf3dcf6ac5d0 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:43.641060 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 976/1947] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:43 2022] POST /identity/v3/users => generated 293 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:43.644054 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980d61af-723d-48de-ac6a-07603d8cfa64 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:43.645580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980d61af-723d-48de-ac6a-07603d8cfa64 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:43.645898 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980d61af-723d-48de-ac6a-07603d8cfa64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:43.657698 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-9c2a78b0-a761-4429-8940-bbd8ee547f05 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a4b986d177014d749891075e7692d902'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:43.669369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-980d61af-723d-48de-ac6a-07603d8cfa64 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:43.670657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-980d61af-723d-48de-ac6a-07603d8cfa64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:43.680061 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 977/1948] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:43.681131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980d61af-723d-48de-ac6a-07603d8cfa64 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:43.692804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980d61af-723d-48de-ac6a-07603d8cfa64 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:43.697819 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8225dbc-46f5-4cc3-99bc-8e6311dc73a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:43.698549 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8225dbc-46f5-4cc3-99bc-8e6311dc73a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:43.699040 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8225dbc-46f5-4cc3-99bc-8e6311dc73a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:43.699417 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8225dbc-46f5-4cc3-99bc-8e6311dc73a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:43.700170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8225dbc-46f5-4cc3-99bc-8e6311dc73a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:43.704170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8225dbc-46f5-4cc3-99bc-8e6311dc73a8 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:43.705140 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-9c2a78b0-a761-4429-8940-bbd8ee547f05 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a4b986d177014d749891075e7692d902'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:43.705934 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-9c2a78b0-a761-4429-8940-bbd8ee547f05 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a4b986d177014d749891075e7692d902'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:43.715512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 978/1949] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:43 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:43.717282 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f557128-d8ef-4317-ac09-b939856d7cf6 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:43.717779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f557128-d8ef-4317-ac09-b939856d7cf6 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:43.718742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f557128-d8ef-4317-ac09-b939856d7cf6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:43.720033 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-9c2a78b0-a761-4429-8940-bbd8ee547f05 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a4b986d177014d749891075e7692d902'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:43.751946 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-9c2a78b0-a761-4429-8940-bbd8ee547f05 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a4b986d177014d749891075e7692d902'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:43.755257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-2f557128-d8ef-4317-ac09-b939856d7cf6 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:43.757307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-2f557128-d8ef-4317-ac09-b939856d7cf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:43.761030 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 972/1950] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:43 2022] DELETE /identity/v3/users/a4b986d177014d749891075e7692d902 => generated 0 bytes in 360 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:43.762331 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2a78b0-a761-4429-8940-bbd8ee547f05 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:43.764592 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-9c2a78b0-a761-4429-8940-bbd8ee547f05 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:43.779387 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 979/1951] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:43.781134 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f557128-d8ef-4317-ac09-b939856d7cf6 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:43.795441 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f557128-d8ef-4317-ac09-b939856d7cf6 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:43.882080 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2a78b0-a761-4429-8940-bbd8ee547f05 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:43.887738 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19916dad-4e49-43f0-b342-bcf3f79c0848 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:43.888413 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19916dad-4e49-43f0-b342-bcf3f79c0848 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:43.888848 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19916dad-4e49-43f0-b342-bcf3f79c0848 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:43.889129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19916dad-4e49-43f0-b342-bcf3f79c0848 admin admin] PATH_INFO: `/v3/projects/a5d68445c296413a86184878a98b563b/users/04f0174445d0410a86de96a190d654cb/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:43.899510 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5366c740-ca9f-4a37-aab8-848ca0da37e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:43.899823 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5366c740-ca9f-4a37-aab8-848ca0da37e0 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:43.900104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5366c740-ca9f-4a37-aab8-848ca0da37e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:43.900374 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5366c740-ca9f-4a37-aab8-848ca0da37e0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:43.901028 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5366c740-ca9f-4a37-aab8-848ca0da37e0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:20:43.901328 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5366c740-ca9f-4a37-aab8-848ca0da37e0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:43.903309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5366c740-ca9f-4a37-aab8-848ca0da37e0 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:43.909461 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19916dad-4e49-43f0-b342-bcf3f79c0848 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a5d68445c296413a86184878a98b563b, user_id=04f0174445d0410a86de96a190d654cb, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:43.912383 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19916dad-4e49-43f0-b342-bcf3f79c0848 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:43.916496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 980/1952] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:20:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:20:43.920573 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5366c740-ca9f-4a37-aab8-848ca0da37e0 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:43.929579 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 973/1953] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:43 2022] PUT /identity/v3/projects/a5d68445c296413a86184878a98b563b/users/04f0174445d0410a86de96a190d654cb/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:43.931079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5366c740-ca9f-4a37-aab8-848ca0da37e0 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:43.933473 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19916dad-4e49-43f0-b342-bcf3f79c0848 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:43.939463 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:44.006615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2752d08-f0a7-4a0d-a20d-90c60c5a38c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:44.007566 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2752d08-f0a7-4a0d-a20d-90c60c5a38c5 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:44.007747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2752d08-f0a7-4a0d-a20d-90c60c5a38c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:44.008288 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2752d08-f0a7-4a0d-a20d-90c60c5a38c5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:44.009234 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2752d08-f0a7-4a0d-a20d-90c60c5a38c5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:44.015677 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2752d08-f0a7-4a0d-a20d-90c60c5a38c5 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:44.060349 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19916dad-4e49-43f0-b342-bcf3f79c0848 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:44.064715 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 981/1954] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:43 2022] POST /identity/v3/projects => generated 365 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:20:44.065621 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2752d08-f0a7-4a0d-a20d-90c60c5a38c5 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:44.124408 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:44.143956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5d3895-9b9a-484f-af76-5769208f1147 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:44.144744 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5d3895-9b9a-484f-af76-5769208f1147 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:44.146300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5d3895-9b9a-484f-af76-5769208f1147 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:44.147084 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5d3895-9b9a-484f-af76-5769208f1147 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:44.148122 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5d3895-9b9a-484f-af76-5769208f1147 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:44.150197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5d3895-9b9a-484f-af76-5769208f1147 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:44.165717 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 974/1955] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:43 2022] GET /identity/v3/roles => generated 1420 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:20:44.166641 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5d3895-9b9a-484f-af76-5769208f1147 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:44.168838 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-7b5d3895-9b9a-484f-af76-5769208f1147 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:44.174929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2752d08-f0a7-4a0d-a20d-90c60c5a38c5 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:44.182293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1138b023-fb92-43d0-a04d-43d06993d1b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:44.182929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1138b023-fb92-43d0-a04d-43d06993d1b1 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:44.183208 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1138b023-fb92-43d0-a04d-43d06993d1b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:44.183484 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1138b023-fb92-43d0-a04d-43d06993d1b1 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:44.184167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1138b023-fb92-43d0-a04d-43d06993d1b1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:44.186233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5d3895-9b9a-484f-af76-5769208f1147 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:44.188275 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1138b023-fb92-43d0-a04d-43d06993d1b1 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:44.192090 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5269071d-e8a2-4e20-8f9e-51e4d0c9e5e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:44.192883 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5269071d-e8a2-4e20-8f9e-51e4d0c9e5e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:44.193225 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5269071d-e8a2-4e20-8f9e-51e4d0c9e5e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:44.193572 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5269071d-e8a2-4e20-8f9e-51e4d0c9e5e5 admin admin] PATH_INFO: `/v3/users/aef9d5924eb44c74ae86013a6d3e6318` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:44.195212 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5269071d-e8a2-4e20-8f9e-51e4d0c9e5e5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aef9d5924eb44c74ae86013a6d3e6318)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:44.197325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5269071d-e8a2-4e20-8f9e-51e4d0c9e5e5 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:44.222343 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 982/1956] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:44 2022] POST /identity/v3/users => generated 283 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:44.223834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1138b023-fb92-43d0-a04d-43d06993d1b1 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:44.247171 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1138b023-fb92-43d0-a04d-43d06993d1b1 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:44.253535 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e1e312-5122-49c7-8f7b-5d86318f08b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:44.254386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e1e312-5122-49c7-8f7b-5d86318f08b3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:44.254669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e1e312-5122-49c7-8f7b-5d86318f08b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:44.254946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e1e312-5122-49c7-8f7b-5d86318f08b3 admin admin] PATH_INFO: `/v3/projects/a5d68445c296413a86184878a98b563b/users/04f0174445d0410a86de96a190d654cb/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:44.257506 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e1e312-5122-49c7-8f7b-5d86318f08b3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a5d68445c296413a86184878a98b563b, user_id=04f0174445d0410a86de96a190d654cb, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:44.260416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e1e312-5122-49c7-8f7b-5d86318f08b3 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:44.269895 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-5269071d-e8a2-4e20-8f9e-51e4d0c9e5e5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aef9d5924eb44c74ae86013a6d3e6318'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:44.274367 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 983/1957] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:44 2022] PUT /identity/v3/projects/a5d68445c296413a86184878a98b563b/users/04f0174445d0410a86de96a190d654cb/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:44.275264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e1e312-5122-49c7-8f7b-5d86318f08b3 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:44.278172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d3e1e312-5122-49c7-8f7b-5d86318f08b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:44.280212 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-5269071d-e8a2-4e20-8f9e-51e4d0c9e5e5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aef9d5924eb44c74ae86013a6d3e6318'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:44.281063 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-5269071d-e8a2-4e20-8f9e-51e4d0c9e5e5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aef9d5924eb44c74ae86013a6d3e6318'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:44.294164 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-5269071d-e8a2-4e20-8f9e-51e4d0c9e5e5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aef9d5924eb44c74ae86013a6d3e6318'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:44.336396 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-5269071d-e8a2-4e20-8f9e-51e4d0c9e5e5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aef9d5924eb44c74ae86013a6d3e6318'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:44.359561 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 975/1958] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:44 2022] DELETE /identity/v3/users/aef9d5924eb44c74ae86013a6d3e6318 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:44.360589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5269071d-e8a2-4e20-8f9e-51e4d0c9e5e5 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:44.376737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-5269071d-e8a2-4e20-8f9e-51e4d0c9e5e5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:44.381295 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e1e312-5122-49c7-8f7b-5d86318f08b3 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:44.432583 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:44.523730 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf78edf1-4e48-437a-aba2-fc4099891c2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:44.524408 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf78edf1-4e48-437a-aba2-fc4099891c2a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:44.524532 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5269071d-e8a2-4e20-8f9e-51e4d0c9e5e5 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:44.524691 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf78edf1-4e48-437a-aba2-fc4099891c2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:44.525288 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf78edf1-4e48-437a-aba2-fc4099891c2a admin admin] PATH_INFO: `/v3/projects/0ec09a0648bd421981a633352221b9dd` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:44.527274 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf78edf1-4e48-437a-aba2-fc4099891c2a admin admin] RBAC: Authorizing `identity:delete_project(project_id=0ec09a0648bd421981a633352221b9dd)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:44.529458 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf78edf1-4e48-437a-aba2-fc4099891c2a admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:44.530330 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a739fefe-67f2-4fd2-9171-0999e449b5ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:44.530941 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a739fefe-67f2-4fd2-9171-0999e449b5ba admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:44.531226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a739fefe-67f2-4fd2-9171-0999e449b5ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:44.531494 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a739fefe-67f2-4fd2-9171-0999e449b5ba admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:44.532138 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a739fefe-67f2-4fd2-9171-0999e449b5ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:44.537797 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a739fefe-67f2-4fd2-9171-0999e449b5ba admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:44.554184 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 976/1959] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:44 2022] GET /identity/v3/roles => generated 1420 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:44.554184 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d29ccd-95e8-4792-aec9-eaa954e9be6e None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:44.554184 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d29ccd-95e8-4792-aec9-eaa954e9be6e None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:44.554184 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d29ccd-95e8-4792-aec9-eaa954e9be6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:44.579003 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-36d29ccd-95e8-4792-aec9-eaa954e9be6e None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:44.580938 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-36d29ccd-95e8-4792-aec9-eaa954e9be6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:44.593245 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-bf78edf1-4e48-437a-aba2-fc4099891c2a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0ec09a0648bd421981a633352221b9dd', 'request_id': 'req-bf78edf1-4e48-437a-aba2-fc4099891c2a'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:44.599730 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-bf78edf1-4e48-437a-aba2-fc4099891c2a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0ec09a0648bd421981a633352221b9dd', 'request_id': 'req-bf78edf1-4e48-437a-aba2-fc4099891c2a'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:44.600963 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=90,keystone:INSERT=4,keystone:DELETE=6 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:44.612348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-bf78edf1-4e48-437a-aba2-fc4099891c2a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0ec09a0648bd421981a633352221b9dd', 'request_id': 'req-bf78edf1-4e48-437a-aba2-fc4099891c2a'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:44.630142 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-bf78edf1-4e48-437a-aba2-fc4099891c2a admin admin] The token cache is being invalidate because project 0ec09a0648bd421981a633352221b9dd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:44.630433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-bf78edf1-4e48-437a-aba2-fc4099891c2a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:44.631571 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-bf78edf1-4e48-437a-aba2-fc4099891c2a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:44.633126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 984/1960] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:44 2022] DELETE /identity/v3/projects/0ec09a0648bd421981a633352221b9dd => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:44.633690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf78edf1-4e48-437a-aba2-fc4099891c2a admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:44.636619 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-bf78edf1-4e48-437a-aba2-fc4099891c2a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:44.666406 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 977/1961] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:44.666406 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5b7061-5710-4549-b164-9dcead5200a7 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:44.666406 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5b7061-5710-4549-b164-9dcead5200a7 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:44.666406 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5b7061-5710-4549-b164-9dcead5200a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:44.690603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-ec5b7061-5710-4549-b164-9dcead5200a7 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:44.691833 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ec5b7061-5710-4549-b164-9dcead5200a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:44.701058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 978/1962] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:44.701902 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5b7061-5710-4549-b164-9dcead5200a7 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:44.707066 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ec5b7061-5710-4549-b164-9dcead5200a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:44.713142 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf78edf1-4e48-437a-aba2-fc4099891c2a admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:44.720046 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84dbe56e-74d2-456f-a184-4d1973e245d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:44.721382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84dbe56e-74d2-456f-a184-4d1973e245d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:44.721861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84dbe56e-74d2-456f-a184-4d1973e245d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:44.722304 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84dbe56e-74d2-456f-a184-4d1973e245d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:44.723104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84dbe56e-74d2-456f-a184-4d1973e245d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:44.724869 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84dbe56e-74d2-456f-a184-4d1973e245d8 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:44.738457 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 985/1963] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:44 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:44.740560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d71cfd-6831-41ef-ad9f-527693e9f667 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:44.741513 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d71cfd-6831-41ef-ad9f-527693e9f667 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:44.742091 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d71cfd-6831-41ef-ad9f-527693e9f667 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:44.768339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-e2d71cfd-6831-41ef-ad9f-527693e9f667 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:44.769672 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-e2d71cfd-6831-41ef-ad9f-527693e9f667 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:44.778766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 986/1964] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:44.779768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d71cfd-6831-41ef-ad9f-527693e9f667 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:44.791327 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d71cfd-6831-41ef-ad9f-527693e9f667 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:44.797144 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245a8ab9-2e5b-49ff-a31c-63e66b84c935 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:44.797823 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245a8ab9-2e5b-49ff-a31c-63e66b84c935 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:44.798169 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245a8ab9-2e5b-49ff-a31c-63e66b84c935 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:44.798512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245a8ab9-2e5b-49ff-a31c-63e66b84c935 admin admin] PATH_INFO: `/v3/projects/8c7092bb4084473fa41466f1f4af3154/users/86f508453722447eab52cfef5f0f181c/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:44.803123 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5b7061-5710-4549-b164-9dcead5200a7 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:44.807847 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdfc1c3c-4ea4-470e-8f65-e16b80641923 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:44.808485 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfc1c3c-4ea4-470e-8f65-e16b80641923 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:44.808767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfc1c3c-4ea4-470e-8f65-e16b80641923 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:44.809123 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfc1c3c-4ea4-470e-8f65-e16b80641923 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:44.821120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245a8ab9-2e5b-49ff-a31c-63e66b84c935 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8c7092bb4084473fa41466f1f4af3154, user_id=86f508453722447eab52cfef5f0f181c, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:44.823864 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245a8ab9-2e5b-49ff-a31c-63e66b84c935 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:44.830551 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdfc1c3c-4ea4-470e-8f65-e16b80641923 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:44.831994 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdfc1c3c-4ea4-470e-8f65-e16b80641923 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:44.844810 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 979/1965] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:44.845597 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdfc1c3c-4ea4-470e-8f65-e16b80641923 service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:44.856939 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 987/1966] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:44 2022] PUT /identity/v3/projects/8c7092bb4084473fa41466f1f4af3154/users/86f508453722447eab52cfef5f0f181c/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:44.871268 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b7a408-7def-4618-9f22-2cdd234572e7 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:44.871268 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b7a408-7def-4618-9f22-2cdd234572e7 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:44.871535 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b7a408-7def-4618-9f22-2cdd234572e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:44.886706 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdfc1c3c-4ea4-470e-8f65-e16b80641923 service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:44.925932 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-73b7a408-7def-4618-9f22-2cdd234572e7 None None] MFA Rules not processed for user `86f508453722447eab52cfef5f0f181c`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:44.927164 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-73b7a408-7def-4618-9f22-2cdd234572e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:45.009179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:45.029128 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ad272e-680d-46e4-8d47-ac882b37ce11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:45.032617 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ad272e-680d-46e4-8d47-ac882b37ce11 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:45.032913 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ad272e-680d-46e4-8d47-ac882b37ce11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:45.033483 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ad272e-680d-46e4-8d47-ac882b37ce11 admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:45.039132 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ad272e-680d-46e4-8d47-ac882b37ce11 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:20:45.039447 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ad272e-680d-46e4-8d47-ac882b37ce11 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:45.046390 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ad272e-680d-46e4-8d47-ac882b37ce11 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:45.073500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 980/1967] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:20:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:20:45.076482 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ad272e-680d-46e4-8d47-ac882b37ce11 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:45.096267 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 988/1968] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:44 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:45.102285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ad272e-680d-46e4-8d47-ac882b37ce11 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:45.102880 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b7a408-7def-4618-9f22-2cdd234572e7 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:45.107706 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5b5d56-5973-4657-9594-88f9cebe9fea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:45.109045 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5b5d56-5973-4657-9594-88f9cebe9fea admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:45.109402 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5b5d56-5973-4657-9594-88f9cebe9fea admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:45.109760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5b5d56-5973-4657-9594-88f9cebe9fea admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:45.111084 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5b5d56-5973-4657-9594-88f9cebe9fea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:45.113285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5b5d56-5973-4657-9594-88f9cebe9fea admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:45.125038 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b7a408-7def-4618-9f22-2cdd234572e7 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:45.134076 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebded964-de64-4fa0-b44a-fd55a348198e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:45.134849 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebded964-de64-4fa0-b44a-fd55a348198e admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:45.135282 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebded964-de64-4fa0-b44a-fd55a348198e admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:45.135794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebded964-de64-4fa0-b44a-fd55a348198e admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:45.136498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebded964-de64-4fa0-b44a-fd55a348198e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:45.138971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebded964-de64-4fa0-b44a-fd55a348198e admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:45.158893 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 981/1969] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:45 2022] POST /identity/v3/projects => generated 399 bytes in 83 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:45.164249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5b5d56-5973-4657-9594-88f9cebe9fea admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:45.181436 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 989/1970] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:45 2022] POST /identity/v3/projects => generated 361 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:20:45.184013 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebded964-de64-4fa0-b44a-fd55a348198e admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:45.224198 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:45.311928 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5b5d56-5973-4657-9594-88f9cebe9fea admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:45.358654 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebded964-de64-4fa0-b44a-fd55a348198e admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:45.363776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08797a6e-3396-49eb-8e29-a0a5fb211b5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:45.364475 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08797a6e-3396-49eb-8e29-a0a5fb211b5c admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:45.364905 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08797a6e-3396-49eb-8e29-a0a5fb211b5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:45.365191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08797a6e-3396-49eb-8e29-a0a5fb211b5c admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:45.365993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08797a6e-3396-49eb-8e29-a0a5fb211b5c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:45.367945 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08797a6e-3396-49eb-8e29-a0a5fb211b5c admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:45.403131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 990/1971] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:45 2022] POST /identity/v3/users => generated 287 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:20:45.404363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08797a6e-3396-49eb-8e29-a0a5fb211b5c admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:45.420189 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-08797a6e-3396-49eb-8e29-a0a5fb211b5c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:45.439331 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08797a6e-3396-49eb-8e29-a0a5fb211b5c admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:45.450831 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915581b9-1629-45eb-a698-1c5bb88195c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:45.451518 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915581b9-1629-45eb-a698-1c5bb88195c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:45.451824 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915581b9-1629-45eb-a698-1c5bb88195c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:45.452599 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915581b9-1629-45eb-a698-1c5bb88195c3 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:45.453602 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915581b9-1629-45eb-a698-1c5bb88195c3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:45.454670 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ada6e84-9dac-41ec-bb62-9884eb77e76e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:45.455853 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915581b9-1629-45eb-a698-1c5bb88195c3 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:45.456058 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ada6e84-9dac-41ec-bb62-9884eb77e76e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:45.456663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ada6e84-9dac-41ec-bb62-9884eb77e76e admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:45.456663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ada6e84-9dac-41ec-bb62-9884eb77e76e admin admin] PATH_INFO: `/v3/users/0b5828afcc48467fa76b308eb8426f64` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:45.458633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ada6e84-9dac-41ec-bb62-9884eb77e76e admin admin] RBAC: Authorizing `identity:delete_user(user_id=0b5828afcc48467fa76b308eb8426f64)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:45.460848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ada6e84-9dac-41ec-bb62-9884eb77e76e admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:45.488654 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 982/1972] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:45 2022] POST /identity/v3/users => generated 300 bytes in 325 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:45.489657 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915581b9-1629-45eb-a698-1c5bb88195c3 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:45.492791 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-915581b9-1629-45eb-a698-1c5bb88195c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:45.509670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915581b9-1629-45eb-a698-1c5bb88195c3 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:45.516427 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb970f4-f7b5-4a90-b1cb-a41fab96cb30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:45.517718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb970f4-f7b5-4a90-b1cb-a41fab96cb30 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:45.518077 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb970f4-f7b5-4a90-b1cb-a41fab96cb30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:45.518503 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb970f4-f7b5-4a90-b1cb-a41fab96cb30 admin admin] PATH_INFO: `/v3/projects/cfe73812e575488ab8984bce1a0246ad` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:45.519787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb970f4-f7b5-4a90-b1cb-a41fab96cb30 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cfe73812e575488ab8984bce1a0246ad)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:45.522599 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb970f4-f7b5-4a90-b1cb-a41fab96cb30 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:45.549310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-4ada6e84-9dac-41ec-bb62-9884eb77e76e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0b5828afcc48467fa76b308eb8426f64'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:45.559338 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-4ada6e84-9dac-41ec-bb62-9884eb77e76e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0b5828afcc48467fa76b308eb8426f64'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:45.561093 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-4ada6e84-9dac-41ec-bb62-9884eb77e76e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0b5828afcc48467fa76b308eb8426f64'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:45.584747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-4ada6e84-9dac-41ec-bb62-9884eb77e76e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0b5828afcc48467fa76b308eb8426f64'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:45.634065 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-4ada6e84-9dac-41ec-bb62-9884eb77e76e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0b5828afcc48467fa76b308eb8426f64'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:45.650840 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 991/1973] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:45 2022] DELETE /identity/v3/users/0b5828afcc48467fa76b308eb8426f64 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:45.651789 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ada6e84-9dac-41ec-bb62-9884eb77e76e admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:45.653455 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-4ada6e84-9dac-41ec-bb62-9884eb77e76e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:45.661901 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0bb970f4-f7b5-4a90-b1cb-a41fab96cb30 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cfe73812e575488ab8984bce1a0246ad', 'request_id': 'req-0bb970f4-f7b5-4a90-b1cb-a41fab96cb30'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:45.675851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=90,keystone:INSERT=4,keystone:DELETE=6 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:45.677510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0bb970f4-f7b5-4a90-b1cb-a41fab96cb30 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cfe73812e575488ab8984bce1a0246ad', 'request_id': 'req-0bb970f4-f7b5-4a90-b1cb-a41fab96cb30'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:45.708578 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0bb970f4-f7b5-4a90-b1cb-a41fab96cb30 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cfe73812e575488ab8984bce1a0246ad', 'request_id': 'req-0bb970f4-f7b5-4a90-b1cb-a41fab96cb30'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:45.743312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0bb970f4-f7b5-4a90-b1cb-a41fab96cb30 admin admin] The token cache is being invalidate because project cfe73812e575488ab8984bce1a0246ad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:45.743788 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0bb970f4-f7b5-4a90-b1cb-a41fab96cb30 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:45.744763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0bb970f4-f7b5-4a90-b1cb-a41fab96cb30 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:45.751909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 983/1974] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:45 2022] DELETE /identity/v3/projects/cfe73812e575488ab8984bce1a0246ad => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:45.752797 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb970f4-f7b5-4a90-b1cb-a41fab96cb30 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:45.770696 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0bb970f4-f7b5-4a90-b1cb-a41fab96cb30 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:45.795558 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ada6e84-9dac-41ec-bb62-9884eb77e76e admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:45.800440 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8553eb1b-2217-44a2-be31-8a38fe966c1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:45.801054 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8553eb1b-2217-44a2-be31-8a38fe966c1e admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:45.801335 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8553eb1b-2217-44a2-be31-8a38fe966c1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:45.801610 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8553eb1b-2217-44a2-be31-8a38fe966c1e admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:45.802835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8553eb1b-2217-44a2-be31-8a38fe966c1e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:45.803894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8553eb1b-2217-44a2-be31-8a38fe966c1e admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:45.823551 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 992/1975] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:45 2022] GET /identity/v3/roles => generated 1420 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:45.825694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0955195d-18a2-498d-8661-bbbce313cbff None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:45.826226 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0955195d-18a2-498d-8661-bbbce313cbff None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:45.827749 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0955195d-18a2-498d-8661-bbbce313cbff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:45.859449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb970f4-f7b5-4a90-b1cb-a41fab96cb30 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:45.864450 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c4ae70-9d58-4dba-aee6-f7bdcb40c7a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:45.865072 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c4ae70-9d58-4dba-aee6-f7bdcb40c7a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:45.865355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c4ae70-9d58-4dba-aee6-f7bdcb40c7a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:45.865629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c4ae70-9d58-4dba-aee6-f7bdcb40c7a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:45.866128 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-0955195d-18a2-498d-8661-bbbce313cbff None None] MFA Rules not processed for user `04f0174445d0410a86de96a190d654cb`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:45.866322 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c4ae70-9d58-4dba-aee6-f7bdcb40c7a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:45.867698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-0955195d-18a2-498d-8661-bbbce313cbff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:45.867914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c4ae70-9d58-4dba-aee6-f7bdcb40c7a3 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:45.873827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:45.882531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 984/1976] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:45 2022] GET /identity/v3/roles => generated 1420 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:45.884355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c7bc2e-3e94-430f-bc90-30e5f5efebaa None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:45.884585 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c7bc2e-3e94-430f-bc90-30e5f5efebaa None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:45.884859 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c7bc2e-3e94-430f-bc90-30e5f5efebaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:45.906447 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-49c7bc2e-3e94-430f-bc90-30e5f5efebaa None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:45.907725 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-49c7bc2e-3e94-430f-bc90-30e5f5efebaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:45.916565 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 985/1977] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:45.918241 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c7bc2e-3e94-430f-bc90-30e5f5efebaa None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:45.921270 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-49c7bc2e-3e94-430f-bc90-30e5f5efebaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:45.936128 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c7bc2e-3e94-430f-bc90-30e5f5efebaa None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:45.942407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a311654f-cdc5-4402-9ea3-9e4478f5d8e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:45.943069 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a311654f-cdc5-4402-9ea3-9e4478f5d8e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:45.943373 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a311654f-cdc5-4402-9ea3-9e4478f5d8e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:45.943653 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a311654f-cdc5-4402-9ea3-9e4478f5d8e7 admin admin] PATH_INFO: `/v3/users/d1edc763424a40cd962f9cd517e1ffee` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:45.944871 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a311654f-cdc5-4402-9ea3-9e4478f5d8e7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d1edc763424a40cd962f9cd517e1ffee)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:45.947339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a311654f-cdc5-4402-9ea3-9e4478f5d8e7 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:45.956684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 993/1978] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:45 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:45.958265 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0955195d-18a2-498d-8661-bbbce313cbff None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:45.960325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-0955195d-18a2-498d-8661-bbbce313cbff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:45.974924 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0955195d-18a2-498d-8661-bbbce313cbff None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:45.981204 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4651a99c-b38d-4969-96b5-e3b21299c115 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:45.982252 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4651a99c-b38d-4969-96b5-e3b21299c115 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:45.982526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4651a99c-b38d-4969-96b5-e3b21299c115 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:45.983990 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4651a99c-b38d-4969-96b5-e3b21299c115 admin admin] PATH_INFO: `/v3/projects/40ed12f0e3414ab393dd77ced315a36b/users/2ff642ed545446a5bd4ea16c56bacb5f/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:46.004522 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4651a99c-b38d-4969-96b5-e3b21299c115 admin admin] RBAC: Authorizing `identity:create_grant(project_id=40ed12f0e3414ab393dd77ced315a36b, user_id=2ff642ed545446a5bd4ea16c56bacb5f, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:46.007551 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a311654f-cdc5-4402-9ea3-9e4478f5d8e7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd1edc763424a40cd962f9cd517e1ffee'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:46.007923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4651a99c-b38d-4969-96b5-e3b21299c115 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:46.016393 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a311654f-cdc5-4402-9ea3-9e4478f5d8e7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd1edc763424a40cd962f9cd517e1ffee'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:46.017269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a311654f-cdc5-4402-9ea3-9e4478f5d8e7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd1edc763424a40cd962f9cd517e1ffee'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:46.023516 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 994/1979] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:45 2022] PUT /identity/v3/projects/40ed12f0e3414ab393dd77ced315a36b/users/2ff642ed545446a5bd4ea16c56bacb5f/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:46.024428 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4651a99c-b38d-4969-96b5-e3b21299c115 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:46.026066 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-4651a99c-b38d-4969-96b5-e3b21299c115 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:46.030237 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a311654f-cdc5-4402-9ea3-9e4478f5d8e7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd1edc763424a40cd962f9cd517e1ffee'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:46.047181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a311654f-cdc5-4402-9ea3-9e4478f5d8e7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd1edc763424a40cd962f9cd517e1ffee'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:46.054550 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 986/1980] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:45 2022] DELETE /identity/v3/users/d1edc763424a40cd962f9cd517e1ffee => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:46.055221 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a311654f-cdc5-4402-9ea3-9e4478f5d8e7 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:46.056718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-a311654f-cdc5-4402-9ea3-9e4478f5d8e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:46.116394 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4651a99c-b38d-4969-96b5-e3b21299c115 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:46.154588 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=87,keystone:INSERT=2,keystone:DELETE=11 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:46.165781 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a311654f-cdc5-4402-9ea3-9e4478f5d8e7 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:46.176163 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914a605d-0db3-4e0c-b946-926dea20609b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:46.177161 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914a605d-0db3-4e0c-b946-926dea20609b service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:46.178316 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914a605d-0db3-4e0c-b946-926dea20609b service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:46.178602 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914a605d-0db3-4e0c-b946-926dea20609b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:46.183190 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-914a605d-0db3-4e0c-b946-926dea20609b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:46.211466 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fabe202-e74a-4a3d-982d-d100f437ae2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:46.212659 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fabe202-e74a-4a3d-982d-d100f437ae2f admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:46.213482 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fabe202-e74a-4a3d-982d-d100f437ae2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:46.214481 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fabe202-e74a-4a3d-982d-d100f437ae2f admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:46.215580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fabe202-e74a-4a3d-982d-d100f437ae2f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:46.218089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fabe202-e74a-4a3d-982d-d100f437ae2f admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:46.245547 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 995/1981] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:46 2022] GET /identity/v3/roles => generated 1420 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:46.245547 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fabe202-e74a-4a3d-982d-d100f437ae2f admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:46.248514 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-1fabe202-e74a-4a3d-982d-d100f437ae2f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:46.294044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:46.316258 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914a605d-0db3-4e0c-b946-926dea20609b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:46.317842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914a605d-0db3-4e0c-b946-926dea20609b service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:46.331089 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 987/1982] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 276 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:46.331894 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914a605d-0db3-4e0c-b946-926dea20609b service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:46.333545 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-914a605d-0db3-4e0c-b946-926dea20609b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:46.345092 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fabe202-e74a-4a3d-982d-d100f437ae2f admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:46.349530 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5555f3dd-86b3-4ea9-9dab-6239fb434ff2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:46.350312 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5555f3dd-86b3-4ea9-9dab-6239fb434ff2 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:46.350743 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914a605d-0db3-4e0c-b946-926dea20609b service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:46.351170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5555f3dd-86b3-4ea9-9dab-6239fb434ff2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:46.351463 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5555f3dd-86b3-4ea9-9dab-6239fb434ff2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:46.353631 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-5555f3dd-86b3-4ea9-9dab-6239fb434ff2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:46.356795 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b446736e-7f23-4728-ba1b-a232891eba56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:46.357494 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b446736e-7f23-4728-ba1b-a232891eba56 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:46.357878 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b446736e-7f23-4728-ba1b-a232891eba56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:46.358820 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b446736e-7f23-4728-ba1b-a232891eba56 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:46.360155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b446736e-7f23-4728-ba1b-a232891eba56 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:46.362722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b446736e-7f23-4728-ba1b-a232891eba56 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:46.380293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 988/1983] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:46 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:46.380293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b446736e-7f23-4728-ba1b-a232891eba56 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:46.394614 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b446736e-7f23-4728-ba1b-a232891eba56 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:46.406332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6f4fd3-45e8-4376-85b7-8998455044a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:46.406332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6f4fd3-45e8-4376-85b7-8998455044a7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:46.406332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6f4fd3-45e8-4376-85b7-8998455044a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:46.406332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6f4fd3-45e8-4376-85b7-8998455044a7 admin admin] PATH_INFO: `/v3/projects/2729666a518f4303a2934950e17ffe57/users/fc47926f087447998c185e3e475589e0/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:46.428455 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6f4fd3-45e8-4376-85b7-8998455044a7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2729666a518f4303a2934950e17ffe57, user_id=fc47926f087447998c185e3e475589e0, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:46.432723 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6f4fd3-45e8-4376-85b7-8998455044a7 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:46.451985 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 989/1984] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:46 2022] PUT /identity/v3/projects/2729666a518f4303a2934950e17ffe57/users/fc47926f087447998c185e3e475589e0/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:46.461593 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6f4fd3-45e8-4376-85b7-8998455044a7 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:46.477655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5555f3dd-86b3-4ea9-9dab-6239fb434ff2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:46.479505 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5555f3dd-86b3-4ea9-9dab-6239fb434ff2 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:46.501667 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 996/1985] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:46 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 258 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:46.503026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b9c523-a233-4f9e-b6a8-6131b1059f60 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:46.503304 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b9c523-a233-4f9e-b6a8-6131b1059f60 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:46.503567 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b9c523-a233-4f9e-b6a8-6131b1059f60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:46.549085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-c9b9c523-a233-4f9e-b6a8-6131b1059f60 None None] MFA Rules not processed for user `fc47926f087447998c185e3e475589e0`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:46.550608 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c9b9c523-a233-4f9e-b6a8-6131b1059f60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:46.640820 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6f4fd3-45e8-4376-85b7-8998455044a7 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:46.648023 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ec3d8d-0df5-4d0a-a533-81396ce37729 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:46.649523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ec3d8d-0df5-4d0a-a533-81396ce37729 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:46.649934 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ec3d8d-0df5-4d0a-a533-81396ce37729 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:46.650283 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ec3d8d-0df5-4d0a-a533-81396ce37729 admin admin] PATH_INFO: `/v3/projects/40ed12f0e3414ab393dd77ced315a36b/users/2ff642ed545446a5bd4ea16c56bacb5f/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:46.652887 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ec3d8d-0df5-4d0a-a533-81396ce37729 admin admin] RBAC: Authorizing `identity:create_grant(project_id=40ed12f0e3414ab393dd77ced315a36b, user_id=2ff642ed545446a5bd4ea16c56bacb5f, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:46.658683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ec3d8d-0df5-4d0a-a533-81396ce37729 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:46.685785 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 990/1986] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:46 2022] PUT /identity/v3/projects/40ed12f0e3414ab393dd77ced315a36b/users/2ff642ed545446a5bd4ea16c56bacb5f/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:46.685785 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ec3d8d-0df5-4d0a-a533-81396ce37729 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:46.685785 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-f0ec3d8d-0df5-4d0a-a533-81396ce37729 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:46.695539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 997/1987] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:46 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:46.703730 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf0dc43-33e5-4b88-b84d-921d6298e1e7 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:46.703946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf0dc43-33e5-4b88-b84d-921d6298e1e7 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:46.704138 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf0dc43-33e5-4b88-b84d-921d6298e1e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:46.748342 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-7cf0dc43-33e5-4b88-b84d-921d6298e1e7 None None] MFA Rules not processed for user `2ff642ed545446a5bd4ea16c56bacb5f`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:46.750180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-7cf0dc43-33e5-4b88-b84d-921d6298e1e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:46.818444 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:46.836417 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:46.838943 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ec3d8d-0df5-4d0a-a533-81396ce37729 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:46.850857 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571d840f-6a64-47dc-8c8f-bef4bc428441 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:46.852856 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571d840f-6a64-47dc-8c8f-bef4bc428441 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:46.853692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571d840f-6a64-47dc-8c8f-bef4bc428441 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:46.855736 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571d840f-6a64-47dc-8c8f-bef4bc428441 admin admin] PATH_INFO: `/v3/projects/37249b979d864db6a2af88e936781380` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:46.859740 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-571d840f-6a64-47dc-8c8f-bef4bc428441 admin admin] RBAC: Authorizing `identity:delete_project(project_id=37249b979d864db6a2af88e936781380)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:46.863736 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-571d840f-6a64-47dc-8c8f-bef4bc428441 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:46.867896 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 998/1988] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:46 2022] POST /identity/v3/auth/tokens => generated 3305 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:46.869168 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf0dc43-33e5-4b88-b84d-921d6298e1e7 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:46.881191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf0dc43-33e5-4b88-b84d-921d6298e1e7 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:46.886663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe12e928-9f14-4cdb-a304-fcb7d720ff35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:46.887380 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe12e928-9f14-4cdb-a304-fcb7d720ff35 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:46.887717 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe12e928-9f14-4cdb-a304-fcb7d720ff35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:46.888053 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe12e928-9f14-4cdb-a304-fcb7d720ff35 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:46.888690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe12e928-9f14-4cdb-a304-fcb7d720ff35 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:46.891649 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe12e928-9f14-4cdb-a304-fcb7d720ff35 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:46.936003 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 999/1989] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:46 2022] POST /identity/v3/projects => generated 401 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:46.936003 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe12e928-9f14-4cdb-a304-fcb7d720ff35 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:46.974405 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-571d840f-6a64-47dc-8c8f-bef4bc428441 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '37249b979d864db6a2af88e936781380', 'request_id': 'req-571d840f-6a64-47dc-8c8f-bef4bc428441'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:47.000078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-571d840f-6a64-47dc-8c8f-bef4bc428441 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '37249b979d864db6a2af88e936781380', 'request_id': 'req-571d840f-6a64-47dc-8c8f-bef4bc428441'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:47.019722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-571d840f-6a64-47dc-8c8f-bef4bc428441 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '37249b979d864db6a2af88e936781380', 'request_id': 'req-571d840f-6a64-47dc-8c8f-bef4bc428441'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:47.032029 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe12e928-9f14-4cdb-a304-fcb7d720ff35 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:47.042719 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-571d840f-6a64-47dc-8c8f-bef4bc428441 admin admin] The token cache is being invalidate because project 37249b979d864db6a2af88e936781380 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:47.043191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-571d840f-6a64-47dc-8c8f-bef4bc428441 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:47.045929 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-571d840f-6a64-47dc-8c8f-bef4bc428441 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:47.051759 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 991/1990] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:46 2022] DELETE /identity/v3/projects/37249b979d864db6a2af88e936781380 => generated 0 bytes in 369 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:47.051759 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571d840f-6a64-47dc-8c8f-bef4bc428441 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:47.051759 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-571d840f-6a64-47dc-8c8f-bef4bc428441 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:47.133412 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557345e4-3005-425e-bf87-2c58acfa86f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:47.134231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557345e4-3005-425e-bf87-2c58acfa86f6 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:47.135078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557345e4-3005-425e-bf87-2c58acfa86f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:47.135426 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557345e4-3005-425e-bf87-2c58acfa86f6 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:47.136183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557345e4-3005-425e-bf87-2c58acfa86f6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:47.138992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557345e4-3005-425e-bf87-2c58acfa86f6 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:47.164357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571d840f-6a64-47dc-8c8f-bef4bc428441 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:47.168727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1000/1991] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:46 2022] POST /identity/v3/users => generated 307 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:47.170354 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dc2c55-7c6c-45da-ba57-5cb1f7ef81bd None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:47.171119 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dc2c55-7c6c-45da-ba57-5cb1f7ef81bd None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:47.171437 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dc2c55-7c6c-45da-ba57-5cb1f7ef81bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:47.179611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf21ce2-898b-4a43-9981-42cad65a164e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:47.180754 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf21ce2-898b-4a43-9981-42cad65a164e service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:47.181075 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf21ce2-898b-4a43-9981-42cad65a164e service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:47.181309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf21ce2-898b-4a43-9981-42cad65a164e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:47.192689 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0bf21ce2-898b-4a43-9981-42cad65a164e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:47.209218 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-03dc2c55-7c6c-45da-ba57-5cb1f7ef81bd None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:47.211300 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-03dc2c55-7c6c-45da-ba57-5cb1f7ef81bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:47.222012 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1001/1992] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:47.222966 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03dc2c55-7c6c-45da-ba57-5cb1f7ef81bd None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:47.224035 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-03dc2c55-7c6c-45da-ba57-5cb1f7ef81bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:47.236780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03dc2c55-7c6c-45da-ba57-5cb1f7ef81bd None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:47.241823 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7264e151-9870-460c-9b84-76f6a25b8b25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:47.242432 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7264e151-9870-460c-9b84-76f6a25b8b25 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:47.242709 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7264e151-9870-460c-9b84-76f6a25b8b25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:47.242981 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7264e151-9870-460c-9b84-76f6a25b8b25 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:47.243955 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7264e151-9870-460c-9b84-76f6a25b8b25 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:47.245367 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7264e151-9870-460c-9b84-76f6a25b8b25 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:47.261667 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1002/1993] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:47 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:47.263094 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de07b94a-4f7b-4e0f-a1ce-5b37b48e5c78 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:47.263509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de07b94a-4f7b-4e0f-a1ce-5b37b48e5c78 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:47.263911 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de07b94a-4f7b-4e0f-a1ce-5b37b48e5c78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:47.286685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-de07b94a-4f7b-4e0f-a1ce-5b37b48e5c78 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:47.287818 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-de07b94a-4f7b-4e0f-a1ce-5b37b48e5c78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:47.298931 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1003/1994] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:47.299839 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de07b94a-4f7b-4e0f-a1ce-5b37b48e5c78 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:47.305819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf21ce2-898b-4a43-9981-42cad65a164e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:47.308797 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf21ce2-898b-4a43-9981-42cad65a164e service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:47.310141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de07b94a-4f7b-4e0f-a1ce-5b37b48e5c78 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:47.316859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5883c913-5d1a-46d1-8790-feacab269f28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:47.318002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5883c913-5d1a-46d1-8790-feacab269f28 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:47.318329 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5883c913-5d1a-46d1-8790-feacab269f28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:47.318678 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5883c913-5d1a-46d1-8790-feacab269f28 admin admin] PATH_INFO: `/v3/projects/88d7c063eb474468a8bf3fbc131c3b0f/users/72490b69a32649a3bea8ef3cd2002f14/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:47.323566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 992/1995] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:47 2022] GET /identity/v3/auth/tokens => generated 3305 bytes in 275 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:47.324665 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf21ce2-898b-4a43-9981-42cad65a164e service cinder] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:47.326424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0bf21ce2-898b-4a43-9981-42cad65a164e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:47.338116 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5883c913-5d1a-46d1-8790-feacab269f28 admin admin] RBAC: Authorizing `identity:create_grant(project_id=88d7c063eb474468a8bf3fbc131c3b0f, user_id=72490b69a32649a3bea8ef3cd2002f14, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:47.341937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5883c913-5d1a-46d1-8790-feacab269f28 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:47.369892 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf21ce2-898b-4a43-9981-42cad65a164e service cinder] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:47.375659 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1004/1996] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:47 2022] PUT /identity/v3/projects/88d7c063eb474468a8bf3fbc131c3b0f/users/72490b69a32649a3bea8ef3cd2002f14/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:47.375659 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5883c913-5d1a-46d1-8790-feacab269f28 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:47.407705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:47.455659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:47.492525 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed706953-ee11-4600-8c84-6aa3ad62106b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:47.493476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed706953-ee11-4600-8c84-6aa3ad62106b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:47.493993 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed706953-ee11-4600-8c84-6aa3ad62106b admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:47.494683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed706953-ee11-4600-8c84-6aa3ad62106b admin admin] PATH_INFO: `/v3/projects/17c0e187271d4aab948bd1d9395d35d3` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:47.496252 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed706953-ee11-4600-8c84-6aa3ad62106b admin admin] RBAC: Authorizing `identity:delete_project(project_id=17c0e187271d4aab948bd1d9395d35d3)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:47.496413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5883c913-5d1a-46d1-8790-feacab269f28 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:47.499780 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed706953-ee11-4600-8c84-6aa3ad62106b admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:47.502813 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ef5994-d5a2-463c-b52d-c303ec44f3bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:47.503711 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ef5994-d5a2-463c-b52d-c303ec44f3bb admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:47.503992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ef5994-d5a2-463c-b52d-c303ec44f3bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:47.504267 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ef5994-d5a2-463c-b52d-c303ec44f3bb admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:47.504983 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ef5994-d5a2-463c-b52d-c303ec44f3bb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:20:47.505284 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ef5994-d5a2-463c-b52d-c303ec44f3bb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:47.507099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ef5994-d5a2-463c-b52d-c303ec44f3bb admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:47.525823 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1005/1997] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:20:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:20:47.525823 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ef5994-d5a2-463c-b52d-c303ec44f3bb admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:47.544416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ef5994-d5a2-463c-b52d-c303ec44f3bb admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:47.551490 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f741bef7-274b-44bf-beb9-07e388d70d73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:47.551685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f741bef7-274b-44bf-beb9-07e388d70d73 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:47.551965 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f741bef7-274b-44bf-beb9-07e388d70d73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:47.552279 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f741bef7-274b-44bf-beb9-07e388d70d73 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:47.552969 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f741bef7-274b-44bf-beb9-07e388d70d73 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:47.554673 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f741bef7-274b-44bf-beb9-07e388d70d73 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:47.566735 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-ed706953-ee11-4600-8c84-6aa3ad62106b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '17c0e187271d4aab948bd1d9395d35d3', 'request_id': 'req-ed706953-ee11-4600-8c84-6aa3ad62106b'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:47.567515 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1006/1998] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:47 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:47.569071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f741bef7-274b-44bf-beb9-07e388d70d73 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:47.576005 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-ed706953-ee11-4600-8c84-6aa3ad62106b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '17c0e187271d4aab948bd1d9395d35d3', 'request_id': 'req-ed706953-ee11-4600-8c84-6aa3ad62106b'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:47.588962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-ed706953-ee11-4600-8c84-6aa3ad62106b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '17c0e187271d4aab948bd1d9395d35d3', 'request_id': 'req-ed706953-ee11-4600-8c84-6aa3ad62106b'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:47.610546 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-ed706953-ee11-4600-8c84-6aa3ad62106b admin admin] The token cache is being invalidate because project 17c0e187271d4aab948bd1d9395d35d3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:47.611182 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-ed706953-ee11-4600-8c84-6aa3ad62106b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:47.612650 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-ed706953-ee11-4600-8c84-6aa3ad62106b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:47.614899 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 993/1999] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:47 2022] DELETE /identity/v3/projects/17c0e187271d4aab948bd1d9395d35d3 => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:47.616731 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed706953-ee11-4600-8c84-6aa3ad62106b admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:47.618237 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ed706953-ee11-4600-8c84-6aa3ad62106b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:47.644968 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f741bef7-274b-44bf-beb9-07e388d70d73 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:47.711657 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed706953-ee11-4600-8c84-6aa3ad62106b admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:47.712627 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d58ce82-0f56-4cd7-9735-fbfb89d8d3fa service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:47.713468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d58ce82-0f56-4cd7-9735-fbfb89d8d3fa service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:47.713679 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d58ce82-0f56-4cd7-9735-fbfb89d8d3fa service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:47.713878 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d58ce82-0f56-4cd7-9735-fbfb89d8d3fa service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:47.716414 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-2d58ce82-0f56-4cd7-9735-fbfb89d8d3fa service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:47.740282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a63d15-1641-418e-9739-4722fdd2ae99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:47.741357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a63d15-1641-418e-9739-4722fdd2ae99 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:47.742097 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a63d15-1641-418e-9739-4722fdd2ae99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:47.742555 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a63d15-1641-418e-9739-4722fdd2ae99 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:47.743423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a63d15-1641-418e-9739-4722fdd2ae99 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:47.745546 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a63d15-1641-418e-9739-4722fdd2ae99 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:47.793211 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d58ce82-0f56-4cd7-9735-fbfb89d8d3fa service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:47.793576 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 994/2000] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:47 2022] POST /identity/v3/projects => generated 369 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:47.795079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d58ce82-0f56-4cd7-9735-fbfb89d8d3fa service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:47.795311 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a63d15-1641-418e-9739-4722fdd2ae99 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:47.799683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-40a63d15-1641-418e-9739-4722fdd2ae99 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:47.806858 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1007/2001] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:47 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 238 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:47.808395 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba2933c-0887-4c18-9d7d-6513e87c447c None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:47.808735 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba2933c-0887-4c18-9d7d-6513e87c447c None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:47.809004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba2933c-0887-4c18-9d7d-6513e87c447c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:47.832743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-6ba2933c-0887-4c18-9d7d-6513e87c447c None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:47.834276 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6ba2933c-0887-4c18-9d7d-6513e87c447c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:47.841777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:47.923834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:47.928889 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1008/2002] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:47.930835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba2933c-0887-4c18-9d7d-6513e87c447c None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:47.947777 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a63d15-1641-418e-9739-4722fdd2ae99 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:47.958795 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b44b90-8a9d-4c54-a6bf-da4ecf3832fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:47.959403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b44b90-8a9d-4c54-a6bf-da4ecf3832fb admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:47.959687 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b44b90-8a9d-4c54-a6bf-da4ecf3832fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:47.959964 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b44b90-8a9d-4c54-a6bf-da4ecf3832fb admin admin] PATH_INFO: `/v3/projects/88d7c063eb474468a8bf3fbc131c3b0f/users/72490b69a32649a3bea8ef3cd2002f14/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:47.962188 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b44b90-8a9d-4c54-a6bf-da4ecf3832fb admin admin] RBAC: Authorizing `identity:create_grant(project_id=88d7c063eb474468a8bf3fbc131c3b0f, user_id=72490b69a32649a3bea8ef3cd2002f14, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:47.964825 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b44b90-8a9d-4c54-a6bf-da4ecf3832fb admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:47.967528 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba2933c-0887-4c18-9d7d-6513e87c447c None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:47.980314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3a1c26-3066-4b91-9a8c-7ccc6613ea04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:47.981190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3a1c26-3066-4b91-9a8c-7ccc6613ea04 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:47.984249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3a1c26-3066-4b91-9a8c-7ccc6613ea04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:47.985325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3a1c26-3066-4b91-9a8c-7ccc6613ea04 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:47.986898 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3a1c26-3066-4b91-9a8c-7ccc6613ea04 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:47.993827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3a1c26-3066-4b91-9a8c-7ccc6613ea04 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:47.993953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 995/2003] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:47 2022] PUT /identity/v3/projects/88d7c063eb474468a8bf3fbc131c3b0f/users/72490b69a32649a3bea8ef3cd2002f14/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:47.995176 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefb5d8e-d81c-440c-a1d2-50642b43a94e None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:47.995467 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefb5d8e-d81c-440c-a1d2-50642b43a94e None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:47.995745 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefb5d8e-d81c-440c-a1d2-50642b43a94e None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:47.999611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 996/2004] 10.222.0.26 () {54 vars in 921 bytes} [Tue Sep 27 04:20:47 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:20:48.000581 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54cae2a-bee4-4716-a8cf-c85db9ffca41 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:48.000862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54cae2a-bee4-4716-a8cf-c85db9ffca41 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:48.001134 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54cae2a-bee4-4716-a8cf-c85db9ffca41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:48.050834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1009/2005] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:47 2022] POST /identity/v3/users => generated 285 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:48.051968 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ed9fed-2b04-4686-8ea5-3356480adcf3 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:48.052268 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ed9fed-2b04-4686-8ea5-3356480adcf3 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:48.052529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ed9fed-2b04-4686-8ea5-3356480adcf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:48.056615 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-c54cae2a-bee4-4716-a8cf-c85db9ffca41 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:48.057895 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c54cae2a-bee4-4716-a8cf-c85db9ffca41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:48.126788 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-42ed9fed-2b04-4686-8ea5-3356480adcf3 None None] MFA Rules not processed for user `72490b69a32649a3bea8ef3cd2002f14`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:48.127336 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-42ed9fed-2b04-4686-8ea5-3356480adcf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:48.221734 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 997/2006] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:48.223973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ffee69-22c7-42df-918b-4424f5b3ef90 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:48.224719 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ffee69-22c7-42df-918b-4424f5b3ef90 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:48.226597 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ffee69-22c7-42df-918b-4424f5b3ef90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:48.259573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-94ffee69-22c7-42df-918b-4424f5b3ef90 None None] MFA Rules not processed for user `a1586ee841cd44658e4d5feb25de7940`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:48.261062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-94ffee69-22c7-42df-918b-4424f5b3ef90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:48.284722 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1010/2007] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:48 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:48.285283 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ed9fed-2b04-4686-8ea5-3356480adcf3 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:48.301153 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ed9fed-2b04-4686-8ea5-3356480adcf3 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:48.316074 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb69cfb-d15f-438a-baff-8f9241672396 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:48.316837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb69cfb-d15f-438a-baff-8f9241672396 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:48.317119 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb69cfb-d15f-438a-baff-8f9241672396 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:48.317390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb69cfb-d15f-438a-baff-8f9241672396 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:48.318067 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb69cfb-d15f-438a-baff-8f9241672396 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:48.319960 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb69cfb-d15f-438a-baff-8f9241672396 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:48.337759 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1011/2008] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:48 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:48.343682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb69cfb-d15f-438a-baff-8f9241672396 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:48.346633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-4eb69cfb-d15f-438a-baff-8f9241672396 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:48.383822 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 998/2009] 10.222.0.26 () {58 vars in 1018 bytes} [Tue Sep 27 04:20:48 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:48.391281 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ffee69-22c7-42df-918b-4424f5b3ef90 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:48.406893 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ffee69-22c7-42df-918b-4424f5b3ef90 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:48.412683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99192293-665f-4a39-96d4-cc8656b0d4f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:48.413152 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99192293-665f-4a39-96d4-cc8656b0d4f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:48.413432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99192293-665f-4a39-96d4-cc8656b0d4f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:48.413704 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99192293-665f-4a39-96d4-cc8656b0d4f8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:48.414426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99192293-665f-4a39-96d4-cc8656b0d4f8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:20:48.414728 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99192293-665f-4a39-96d4-cc8656b0d4f8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:48.416704 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99192293-665f-4a39-96d4-cc8656b0d4f8 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:48.438982 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 999/2010] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:20:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:20:48.438982 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99192293-665f-4a39-96d4-cc8656b0d4f8 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:48.455583 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99192293-665f-4a39-96d4-cc8656b0d4f8 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:48.461948 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f63de9e-d5e7-4260-845a-427fc740c2ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:48.462696 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f63de9e-d5e7-4260-845a-427fc740c2ab admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:48.462988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f63de9e-d5e7-4260-845a-427fc740c2ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:48.463316 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f63de9e-d5e7-4260-845a-427fc740c2ab admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:48.463971 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f63de9e-d5e7-4260-845a-427fc740c2ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:48.465933 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f63de9e-d5e7-4260-845a-427fc740c2ab admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:48.475443 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb69cfb-d15f-438a-baff-8f9241672396 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:48.480365 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5a376d-7e87-4ee8-88a8-004595e80245 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:48.481088 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5a376d-7e87-4ee8-88a8-004595e80245 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:48.481439 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5a376d-7e87-4ee8-88a8-004595e80245 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:48.481821 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5a376d-7e87-4ee8-88a8-004595e80245 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:48.485860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1000/2011] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:48 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:48.486742 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f63de9e-d5e7-4260-845a-427fc740c2ab admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:48.487776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-4f5a376d-7e87-4ee8-88a8-004595e80245 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:48.488641 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-6f63de9e-d5e7-4260-845a-427fc740c2ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:48.504607 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f63de9e-d5e7-4260-845a-427fc740c2ab admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:48.509871 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e685e9e8-9f66-453b-a9d6-06522e098745 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:48.510971 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e685e9e8-9f66-453b-a9d6-06522e098745 service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:48.511414 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e685e9e8-9f66-453b-a9d6-06522e098745 service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:48.511845 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e685e9e8-9f66-453b-a9d6-06522e098745 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:48.518226 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:48.534272 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e685e9e8-9f66-453b-a9d6-06522e098745 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:48.537176 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e685e9e8-9f66-453b-a9d6-06522e098745 service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:48.554768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1001/2012] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:48 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 68 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:48.556406 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e685e9e8-9f66-453b-a9d6-06522e098745 service cinder] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:48.558259 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-e685e9e8-9f66-453b-a9d6-06522e098745 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:48.573076 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f5a376d-7e87-4ee8-88a8-004595e80245 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:48.574778 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f5a376d-7e87-4ee8-88a8-004595e80245 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:48.579257 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e685e9e8-9f66-453b-a9d6-06522e098745 service cinder] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:48.587452 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c881a3-bc22-4af2-b172-1a505b13e736 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:48.587680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1012/2013] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:48 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 249 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:48.587843 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c881a3-bc22-4af2-b172-1a505b13e736 service nova] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:48.588180 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c881a3-bc22-4af2-b172-1a505b13e736 service nova] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:48.588454 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c881a3-bc22-4af2-b172-1a505b13e736 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:48.599572 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c881a3-bc22-4af2-b172-1a505b13e736 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:48.600528 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5a376d-7e87-4ee8-88a8-004595e80245 service cinder] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:48.601322 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c881a3-bc22-4af2-b172-1a505b13e736 service nova] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:48.616567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1002/2014] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:48 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:48.620940 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c881a3-bc22-4af2-b172-1a505b13e736 service nova] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:48.645418 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5a376d-7e87-4ee8-88a8-004595e80245 service cinder] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:48.651079 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c881a3-bc22-4af2-b172-1a505b13e736 service nova] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:48.663924 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15335458-cf4d-4ef4-b928-41160a7aebf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:48.664136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7021a9-054d-4053-a80e-9d5ec70d8f7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:48.664560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15335458-cf4d-4ef4-b928-41160a7aebf7 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:48.664841 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15335458-cf4d-4ef4-b928-41160a7aebf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:48.665113 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15335458-cf4d-4ef4-b928-41160a7aebf7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:48.665471 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7021a9-054d-4053-a80e-9d5ec70d8f7e admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:48.666036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7021a9-054d-4053-a80e-9d5ec70d8f7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:48.666456 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7021a9-054d-4053-a80e-9d5ec70d8f7e admin admin] PATH_INFO: `/v3/projects/a0c3265513614f2897b64a76b93b7e87/users/ac275ebbabeb46018627ff0259450acf/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:48.673218 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15335458-cf4d-4ef4-b928-41160a7aebf7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:48.675211 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15335458-cf4d-4ef4-b928-41160a7aebf7 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:48.692437 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7021a9-054d-4053-a80e-9d5ec70d8f7e admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0c3265513614f2897b64a76b93b7e87, user_id=ac275ebbabeb46018627ff0259450acf, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:48.695450 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7021a9-054d-4053-a80e-9d5ec70d8f7e admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:48.721053 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1003/2015] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:48 2022] PUT /identity/v3/projects/a0c3265513614f2897b64a76b93b7e87/users/ac275ebbabeb46018627ff0259450acf/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:48.722333 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7021a9-054d-4053-a80e-9d5ec70d8f7e admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:48.740565 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:48.756571 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1013/2016] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:48 2022] POST /identity/v3/projects => generated 383 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:48.758110 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dae80c-f3e6-4b7f-b52f-5c2658e47068 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:48.758399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dae80c-f3e6-4b7f-b52f-5c2658e47068 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:48.758828 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dae80c-f3e6-4b7f-b52f-5c2658e47068 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:48.793777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-99dae80c-f3e6-4b7f-b52f-5c2658e47068 None None] MFA Rules not processed for user `ac275ebbabeb46018627ff0259450acf`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:48.795067 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-99dae80c-f3e6-4b7f-b52f-5c2658e47068 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:48.867198 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7021a9-054d-4053-a80e-9d5ec70d8f7e admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:48.912519 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1014/2017] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:48 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:48.912519 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99dae80c-f3e6-4b7f-b52f-5c2658e47068 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:48.967422 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7ced25-9858-4655-b43a-731d441afa7e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:48.968246 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7ced25-9858-4655-b43a-731d441afa7e service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:48.968360 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7ced25-9858-4655-b43a-731d441afa7e service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:48.968764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7ced25-9858-4655-b43a-731d441afa7e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:48.973119 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ae7ced25-9858-4655-b43a-731d441afa7e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:49.034348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99dae80c-f3e6-4b7f-b52f-5c2658e47068 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:49.040443 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b617c37-2243-4b35-ae0e-d5a123bd4ab6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:49.040443 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b617c37-2243-4b35-ae0e-d5a123bd4ab6 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:49.040443 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b617c37-2243-4b35-ae0e-d5a123bd4ab6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:49.040902 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b617c37-2243-4b35-ae0e-d5a123bd4ab6 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:49.041453 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b617c37-2243-4b35-ae0e-d5a123bd4ab6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:49.044584 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b617c37-2243-4b35-ae0e-d5a123bd4ab6 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:49.074315 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1015/2018] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:48 2022] POST /identity/v3/users => generated 292 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:49.076187 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b617c37-2243-4b35-ae0e-d5a123bd4ab6 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:49.080703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-0b617c37-2243-4b35-ae0e-d5a123bd4ab6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:49.082962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7ced25-9858-4655-b43a-731d441afa7e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:49.084480 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7ced25-9858-4655-b43a-731d441afa7e service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:49.099348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1004/2019] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:48 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:49.100177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7ced25-9858-4655-b43a-731d441afa7e service cinder] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:49.118428 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7ced25-9858-4655-b43a-731d441afa7e service cinder] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:49.137412 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae0ec42-0fd4-4e7e-bca2-c2dcf6831034 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:49.138100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae0ec42-0fd4-4e7e-bca2-c2dcf6831034 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:49.138391 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae0ec42-0fd4-4e7e-bca2-c2dcf6831034 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:49.138831 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae0ec42-0fd4-4e7e-bca2-c2dcf6831034 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:49.140202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae0ec42-0fd4-4e7e-bca2-c2dcf6831034 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:49.142491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae0ec42-0fd4-4e7e-bca2-c2dcf6831034 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:49.146194 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:49.159627 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1005/2020] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:49 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:49.163510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae0ec42-0fd4-4e7e-bca2-c2dcf6831034 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:49.176628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae0ec42-0fd4-4e7e-bca2-c2dcf6831034 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:49.183821 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c2acde-47db-4f4d-a3f1-6fa6377bf0cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:49.186356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c2acde-47db-4f4d-a3f1-6fa6377bf0cf admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:49.187708 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c2acde-47db-4f4d-a3f1-6fa6377bf0cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:49.188336 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c2acde-47db-4f4d-a3f1-6fa6377bf0cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:49.189419 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c2acde-47db-4f4d-a3f1-6fa6377bf0cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:49.191850 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c2acde-47db-4f4d-a3f1-6fa6377bf0cf admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:49.207790 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1006/2021] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:49 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:20:49.218700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c2acde-47db-4f4d-a3f1-6fa6377bf0cf admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:49.249871 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c2acde-47db-4f4d-a3f1-6fa6377bf0cf admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:49.256279 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b617c37-2243-4b35-ae0e-d5a123bd4ab6 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:49.256915 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d941775-14b9-47c7-ac82-925a113418e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:49.257668 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d941775-14b9-47c7-ac82-925a113418e0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:49.258263 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d941775-14b9-47c7-ac82-925a113418e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:49.258685 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d941775-14b9-47c7-ac82-925a113418e0 admin admin] PATH_INFO: `/v3/projects/3690b5efde2e4646ae992f5599eadb2f/users/fd3c3da76ddf4283a1bf5bcb8b606322/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:49.260953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a97ce53-a388-4e10-a144-09e5c756342a service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dcc53332e6b462baf349a72b19d2b3b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:49.261572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a97ce53-a388-4e10-a144-09e5c756342a service glance] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:49.262028 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a97ce53-a388-4e10-a144-09e5c756342a service glance] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:49.262363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a97ce53-a388-4e10-a144-09e5c756342a service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:49.273497 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a97ce53-a388-4e10-a144-09e5c756342a service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:49.275496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a97ce53-a388-4e10-a144-09e5c756342a service glance] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:49.277843 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d941775-14b9-47c7-ac82-925a113418e0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3690b5efde2e4646ae992f5599eadb2f, user_id=fd3c3da76ddf4283a1bf5bcb8b606322, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:49.282198 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d941775-14b9-47c7-ac82-925a113418e0 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:49.294112 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1016/2022] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:49 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:49.304330 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1007/2023] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:49 2022] PUT /identity/v3/projects/3690b5efde2e4646ae992f5599eadb2f/users/fd3c3da76ddf4283a1bf5bcb8b606322/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:49.317576 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631d615b-0728-45cf-a5c2-649ab4097ae5 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:49.317831 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631d615b-0728-45cf-a5c2-649ab4097ae5 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:49.318101 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631d615b-0728-45cf-a5c2-649ab4097ae5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:49.371048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-631d615b-0728-45cf-a5c2-649ab4097ae5 None None] MFA Rules not processed for user `fd3c3da76ddf4283a1bf5bcb8b606322`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:49.380661 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-631d615b-0728-45cf-a5c2-649ab4097ae5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:49.670752 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d941775-14b9-47c7-ac82-925a113418e0 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:49.671104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1017/2024] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:49 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 355 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:20:49.727110 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:49.821989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d941775-14b9-47c7-ac82-925a113418e0 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:49.838160 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b176ce-133f-43dc-bd4c-418b1b9ddc02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:49.842117 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b176ce-133f-43dc-bd4c-418b1b9ddc02 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:49.843329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b176ce-133f-43dc-bd4c-418b1b9ddc02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:49.846111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b176ce-133f-43dc-bd4c-418b1b9ddc02 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:49.847170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b176ce-133f-43dc-bd4c-418b1b9ddc02 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:49.854632 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b176ce-133f-43dc-bd4c-418b1b9ddc02 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:49.920033 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1008/2025] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:49 2022] POST /identity/v3/projects => generated 385 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:20:49.926727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631d615b-0728-45cf-a5c2-649ab4097ae5 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:49.962940 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631d615b-0728-45cf-a5c2-649ab4097ae5 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:50.155177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8912238-a475-49ec-af75-bed4d38bb2f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:50.156216 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8912238-a475-49ec-af75-bed4d38bb2f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:50.156651 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8912238-a475-49ec-af75-bed4d38bb2f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:50.157234 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8912238-a475-49ec-af75-bed4d38bb2f3 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:50.159663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8912238-a475-49ec-af75-bed4d38bb2f3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:50.164416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8912238-a475-49ec-af75-bed4d38bb2f3 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:50.228361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1018/2026] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:49 2022] POST /identity/v3/users => generated 293 bytes in 303 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:50.230455 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b176ce-133f-43dc-bd4c-418b1b9ddc02 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:50.243163 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b176ce-133f-43dc-bd4c-418b1b9ddc02 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:50.249475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4ab91f-92dc-41b0-a92f-7644c9b00db7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:50.250163 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4ab91f-92dc-41b0-a92f-7644c9b00db7 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:50.250739 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4ab91f-92dc-41b0-a92f-7644c9b00db7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:50.250739 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4ab91f-92dc-41b0-a92f-7644c9b00db7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:50.251358 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4ab91f-92dc-41b0-a92f-7644c9b00db7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:50.252980 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4ab91f-92dc-41b0-a92f-7644c9b00db7 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:50.269920 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1009/2027] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:50 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:50.273898 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8912238-a475-49ec-af75-bed4d38bb2f3 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:50.289716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8912238-a475-49ec-af75-bed4d38bb2f3 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:50.317235 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aee2f1d-8935-4862-997b-f739b3ef7b18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:50.318365 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aee2f1d-8935-4862-997b-f739b3ef7b18 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:50.318971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aee2f1d-8935-4862-997b-f739b3ef7b18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:50.319811 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aee2f1d-8935-4862-997b-f739b3ef7b18 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:50.320888 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aee2f1d-8935-4862-997b-f739b3ef7b18 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:50.322693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aee2f1d-8935-4862-997b-f739b3ef7b18 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:50.347718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1019/2028] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:50 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:50.349244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4ab91f-92dc-41b0-a92f-7644c9b00db7 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:50.364832 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4ab91f-92dc-41b0-a92f-7644c9b00db7 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:50.377206 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48e9732-7cee-4691-8cb2-0b44cc4f4b4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:50.378065 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48e9732-7cee-4691-8cb2-0b44cc4f4b4f admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:50.378353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48e9732-7cee-4691-8cb2-0b44cc4f4b4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:50.378692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48e9732-7cee-4691-8cb2-0b44cc4f4b4f admin admin] PATH_INFO: `/v3/projects/0716bfc7dc944ea0804a8631b5b066e0/users/24e32354b9904dca9cc7898c87a03d6f/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:50.419709 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48e9732-7cee-4691-8cb2-0b44cc4f4b4f admin admin] RBAC: Authorizing `identity:create_grant(project_id=0716bfc7dc944ea0804a8631b5b066e0, user_id=24e32354b9904dca9cc7898c87a03d6f, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:50.437272 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48e9732-7cee-4691-8cb2-0b44cc4f4b4f admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:50.499886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1010/2029] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:50 2022] PUT /identity/v3/projects/0716bfc7dc944ea0804a8631b5b066e0/users/24e32354b9904dca9cc7898c87a03d6f/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:50.510788 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab99a060-ec35-4fc9-a006-a78174fe1f11 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:50.511124 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab99a060-ec35-4fc9-a006-a78174fe1f11 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:50.511343 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab99a060-ec35-4fc9-a006-a78174fe1f11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:50.538853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:50.577626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-ab99a060-ec35-4fc9-a006-a78174fe1f11 None None] MFA Rules not processed for user `24e32354b9904dca9cc7898c87a03d6f`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:50.579115 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ab99a060-ec35-4fc9-a006-a78174fe1f11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:50.689791 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1020/2030] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:50 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:50.693250 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48e9732-7cee-4691-8cb2-0b44cc4f4b4f admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:50.930194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48e9732-7cee-4691-8cb2-0b44cc4f4b4f admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:50.938271 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fb2c35-ae60-4c5f-a9a6-e66293214c7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:50.938999 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fb2c35-ae60-4c5f-a9a6-e66293214c7a admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:50.939426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fb2c35-ae60-4c5f-a9a6-e66293214c7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:50.939752 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fb2c35-ae60-4c5f-a9a6-e66293214c7a admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:50.940484 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6fb2c35-ae60-4c5f-a9a6-e66293214c7a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:50.942803 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6fb2c35-ae60-4c5f-a9a6-e66293214c7a admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:50.982602 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1011/2031] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:50 2022] POST /identity/v3/projects => generated 385 bytes in 290 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:50.987702 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab99a060-ec35-4fc9-a006-a78174fe1f11 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:51.006525 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab99a060-ec35-4fc9-a006-a78174fe1f11 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:51.087800 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf91518-f75e-4a9a-a7c1-6ebec8e0d709 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:51.088842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf91518-f75e-4a9a-a7c1-6ebec8e0d709 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:51.088842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf91518-f75e-4a9a-a7c1-6ebec8e0d709 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:51.089042 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf91518-f75e-4a9a-a7c1-6ebec8e0d709 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:51.089816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf91518-f75e-4a9a-a7c1-6ebec8e0d709 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:51.091789 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf91518-f75e-4a9a-a7c1-6ebec8e0d709 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:51.125182 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1021/2032] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:50 2022] POST /identity/v3/users => generated 299 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:51.129064 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fb2c35-ae60-4c5f-a9a6-e66293214c7a admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:51.142484 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fb2c35-ae60-4c5f-a9a6-e66293214c7a admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:51.154973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c09388-105c-4914-a900-8ff5736a31e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:51.156311 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c09388-105c-4914-a900-8ff5736a31e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:51.156311 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c09388-105c-4914-a900-8ff5736a31e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:51.156559 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c09388-105c-4914-a900-8ff5736a31e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:51.157291 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c09388-105c-4914-a900-8ff5736a31e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:51.159140 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c09388-105c-4914-a900-8ff5736a31e1 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:51.173529 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1012/2033] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:51 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:51.179223 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf91518-f75e-4a9a-a7c1-6ebec8e0d709 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:51.189990 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf91518-f75e-4a9a-a7c1-6ebec8e0d709 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:51.195126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d600ff-8656-4ff5-b90a-d71615a3272a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:51.195926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d600ff-8656-4ff5-b90a-d71615a3272a admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:51.196239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d600ff-8656-4ff5-b90a-d71615a3272a admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:51.196588 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d600ff-8656-4ff5-b90a-d71615a3272a admin admin] PATH_INFO: `/v3/projects/a0b62143bb2c44e39402a5483e0c75d2/users/7c091688c92e4a5683e3ac17944f03fa/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:51.215823 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d600ff-8656-4ff5-b90a-d71615a3272a admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0b62143bb2c44e39402a5483e0c75d2, user_id=7c091688c92e4a5683e3ac17944f03fa, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:51.219665 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d600ff-8656-4ff5-b90a-d71615a3272a admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:51.235962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1022/2034] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:51 2022] PUT /identity/v3/projects/a0b62143bb2c44e39402a5483e0c75d2/users/7c091688c92e4a5683e3ac17944f03fa/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:51.239931 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c09388-105c-4914-a900-8ff5736a31e1 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:51.253109 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c09388-105c-4914-a900-8ff5736a31e1 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:51.337236 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:51.349996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e08fe84-dc74-48e1-b77e-cbe37dc968f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:51.350621 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e08fe84-dc74-48e1-b77e-cbe37dc968f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:51.350904 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e08fe84-dc74-48e1-b77e-cbe37dc968f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:51.351205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e08fe84-dc74-48e1-b77e-cbe37dc968f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:51.352496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e08fe84-dc74-48e1-b77e-cbe37dc968f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:51.356582 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e08fe84-dc74-48e1-b77e-cbe37dc968f1 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:51.373971 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1013/2035] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:51 2022] GET /identity/v3/roles => generated 1420 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:20:51.376756 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d600ff-8656-4ff5-b90a-d71615a3272a admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:51.390020 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d600ff-8656-4ff5-b90a-d71615a3272a admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:51.405912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9369931b-43ee-41ec-b5db-b3d981248b6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:51.406632 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9369931b-43ee-41ec-b5db-b3d981248b6d admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:51.406910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9369931b-43ee-41ec-b5db-b3d981248b6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:51.413365 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9369931b-43ee-41ec-b5db-b3d981248b6d admin admin] PATH_INFO: `/v3/projects/a0b62143bb2c44e39402a5483e0c75d2/users/7c091688c92e4a5683e3ac17944f03fa/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:51.416151 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9369931b-43ee-41ec-b5db-b3d981248b6d admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0b62143bb2c44e39402a5483e0c75d2, user_id=7c091688c92e4a5683e3ac17944f03fa, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:51.424935 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9369931b-43ee-41ec-b5db-b3d981248b6d admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:51.443215 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1023/2036] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:51 2022] PUT /identity/v3/projects/a0b62143bb2c44e39402a5483e0c75d2/users/7c091688c92e4a5683e3ac17944f03fa/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:51.457461 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce2f70b-cb51-4f38-a0c2-285fd9822538 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:51.457851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce2f70b-cb51-4f38-a0c2-285fd9822538 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:51.458193 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce2f70b-cb51-4f38-a0c2-285fd9822538 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:51.492608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-8ce2f70b-cb51-4f38-a0c2-285fd9822538 None None] MFA Rules not processed for user `7c091688c92e4a5683e3ac17944f03fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:51.494176 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-8ce2f70b-cb51-4f38-a0c2-285fd9822538 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:51.588274 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1014/2037] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:51 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:51.600066 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9369931b-43ee-41ec-b5db-b3d981248b6d admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:51.670471 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:51.678203 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9369931b-43ee-41ec-b5db-b3d981248b6d admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:51.682076 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff9ffe2-f162-42bf-94e5-8a2e187394dc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:51.682702 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff9ffe2-f162-42bf-94e5-8a2e187394dc service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:51.682991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff9ffe2-f162-42bf-94e5-8a2e187394dc service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:51.683248 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff9ffe2-f162-42bf-94e5-8a2e187394dc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:51.757274 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff9ffe2-f162-42bf-94e5-8a2e187394dc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:51.758764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff9ffe2-f162-42bf-94e5-8a2e187394dc service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:51.768547 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1024/2038] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:51 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 169 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:53.354902 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce2f70b-cb51-4f38-a0c2-285fd9822538 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:53.364764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce2f70b-cb51-4f38-a0c2-285fd9822538 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:53.368339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bf3cd9-a5de-4655-8edf-a1eb0036c03f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:53.368937 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bf3cd9-a5de-4655-8edf-a1eb0036c03f service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:53.369214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bf3cd9-a5de-4655-8edf-a1eb0036c03f service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:53.369495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bf3cd9-a5de-4655-8edf-a1eb0036c03f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:53.436075 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bf3cd9-a5de-4655-8edf-a1eb0036c03f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:53.437654 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bf3cd9-a5de-4655-8edf-a1eb0036c03f service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:53.449763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1015/2039] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:53 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:54.018810 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff9ffe2-f162-42bf-94e5-8a2e187394dc service cinder] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:54.085272 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bf3cd9-a5de-4655-8edf-a1eb0036c03f service cinder] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:54.086848 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-13bf3cd9-a5de-4655-8edf-a1eb0036c03f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:54.099729 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff9ffe2-f162-42bf-94e5-8a2e187394dc service cinder] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:54.104351 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b29976-7edf-4da7-a70d-13766d5324f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:54.105051 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b29976-7edf-4da7-a70d-13766d5324f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:54.105403 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b29976-7edf-4da7-a70d-13766d5324f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:54.105765 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b29976-7edf-4da7-a70d-13766d5324f8 admin admin] PATH_INFO: `/v3/users/fc47926f087447998c185e3e475589e0` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:54.107038 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b29976-7edf-4da7-a70d-13766d5324f8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fc47926f087447998c185e3e475589e0)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:54.107150 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bf3cd9-a5de-4655-8edf-a1eb0036c03f service cinder] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:54.109191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b29976-7edf-4da7-a70d-13766d5324f8 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:54.111927 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20977ddd-2fb6-45d1-9b26-349daa10a712 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:54.112585 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20977ddd-2fb6-45d1-9b26-349daa10a712 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:54.112863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20977ddd-2fb6-45d1-9b26-349daa10a712 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:54.113432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20977ddd-2fb6-45d1-9b26-349daa10a712 admin admin] PATH_INFO: `/v3/users/86f508453722447eab52cfef5f0f181c` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:54.114889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20977ddd-2fb6-45d1-9b26-349daa10a712 admin admin] RBAC: Authorizing `identity:delete_user(user_id=86f508453722447eab52cfef5f0f181c)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:54.116968 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20977ddd-2fb6-45d1-9b26-349daa10a712 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:54.170255 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-07b29976-7edf-4da7-a70d-13766d5324f8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fc47926f087447998c185e3e475589e0'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:54.178922 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-07b29976-7edf-4da7-a70d-13766d5324f8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fc47926f087447998c185e3e475589e0'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:54.179702 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-07b29976-7edf-4da7-a70d-13766d5324f8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fc47926f087447998c185e3e475589e0'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:54.190231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-07b29976-7edf-4da7-a70d-13766d5324f8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fc47926f087447998c185e3e475589e0'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:54.192276 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-20977ddd-2fb6-45d1-9b26-349daa10a712 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '86f508453722447eab52cfef5f0f181c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:54.199933 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-20977ddd-2fb6-45d1-9b26-349daa10a712 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '86f508453722447eab52cfef5f0f181c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:54.200594 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-20977ddd-2fb6-45d1-9b26-349daa10a712 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '86f508453722447eab52cfef5f0f181c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:54.203006 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-07b29976-7edf-4da7-a70d-13766d5324f8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fc47926f087447998c185e3e475589e0'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:54.208383 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-20977ddd-2fb6-45d1-9b26-349daa10a712 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '86f508453722447eab52cfef5f0f181c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:54.209408 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1025/2040] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:54 2022] DELETE /identity/v3/users/fc47926f087447998c185e3e475589e0 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:54.210245 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b29976-7edf-4da7-a70d-13766d5324f8 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:54.213868 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:54.214146 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-07b29976-7edf-4da7-a70d-13766d5324f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:54.228267 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-20977ddd-2fb6-45d1-9b26-349daa10a712 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '86f508453722447eab52cfef5f0f181c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:54.237118 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1016/2041] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:54 2022] DELETE /identity/v3/users/86f508453722447eab52cfef5f0f181c => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:20:54.238671 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a922c3f6-8b0f-4dfa-8fb5-1c40879bfb02 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:54.239193 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a922c3f6-8b0f-4dfa-8fb5-1c40879bfb02 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:54.239539 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a922c3f6-8b0f-4dfa-8fb5-1c40879bfb02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:54.252510 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:54.264946 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-a922c3f6-8b0f-4dfa-8fb5-1c40879bfb02 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:54.266457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-a922c3f6-8b0f-4dfa-8fb5-1c40879bfb02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:54.287451 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b29976-7edf-4da7-a70d-13766d5324f8 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:54.328783 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1017/2042] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:54.332124 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0081742f-57e5-49fc-b492-771c869b71c2 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:54.332124 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0081742f-57e5-49fc-b492-771c869b71c2 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:54.332124 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0081742f-57e5-49fc-b492-771c869b71c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:54.347251 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1c4951-8480-4056-856e-e58880384926 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:54.348155 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1c4951-8480-4056-856e-e58880384926 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:54.348432 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1c4951-8480-4056-856e-e58880384926 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:54.348726 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1c4951-8480-4056-856e-e58880384926 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:54.350473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-0a1c4951-8480-4056-856e-e58880384926 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:54.352207 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-0081742f-57e5-49fc-b492-771c869b71c2 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:54.353383 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0081742f-57e5-49fc-b492-771c869b71c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:54.361456 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1018/2043] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:54.362298 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0081742f-57e5-49fc-b492-771c869b71c2 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:54.363631 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0081742f-57e5-49fc-b492-771c869b71c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:54.417806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1c4951-8480-4056-856e-e58880384926 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:54.418159 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1c4951-8480-4056-856e-e58880384926 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:54.428668 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1026/2044] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:54 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 219 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:54.429426 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1c4951-8480-4056-856e-e58880384926 service cinder] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:54.437864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0081742f-57e5-49fc-b492-771c869b71c2 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:54.439501 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1c4951-8480-4056-856e-e58880384926 service cinder] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:54.442316 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46364e1-1016-4f2d-86e1-4558b80f23d9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:54.443004 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46364e1-1016-4f2d-86e1-4558b80f23d9 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:54.443317 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704a7bf7-9785-43ce-bbee-ee04bfbce074 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:54.443514 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46364e1-1016-4f2d-86e1-4558b80f23d9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:54.443875 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46364e1-1016-4f2d-86e1-4558b80f23d9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:54.444099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704a7bf7-9785-43ce-bbee-ee04bfbce074 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:54.444458 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704a7bf7-9785-43ce-bbee-ee04bfbce074 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:54.444797 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704a7bf7-9785-43ce-bbee-ee04bfbce074 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:54.454547 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704a7bf7-9785-43ce-bbee-ee04bfbce074 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:54.455578 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46364e1-1016-4f2d-86e1-4558b80f23d9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:54.456034 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704a7bf7-9785-43ce-bbee-ee04bfbce074 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:54.457439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46364e1-1016-4f2d-86e1-4558b80f23d9 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:54.468593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1027/2045] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:54.473586 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1019/2046] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:55.149628 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704a7bf7-9785-43ce-bbee-ee04bfbce074 service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:55.151835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-704a7bf7-9785-43ce-bbee-ee04bfbce074 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:55.161690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46364e1-1016-4f2d-86e1-4558b80f23d9 service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:55.163585 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c46364e1-1016-4f2d-86e1-4558b80f23d9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:55.171279 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704a7bf7-9785-43ce-bbee-ee04bfbce074 service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:55.177671 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698fc205-f236-4f54-930f-9fa29f5731df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:55.178559 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698fc205-f236-4f54-930f-9fa29f5731df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:55.178865 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698fc205-f236-4f54-930f-9fa29f5731df admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:55.179142 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698fc205-f236-4f54-930f-9fa29f5731df admin admin] PATH_INFO: `/v3/projects/2729666a518f4303a2934950e17ffe57` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:55.180382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698fc205-f236-4f54-930f-9fa29f5731df admin admin] RBAC: Authorizing `identity:delete_project(project_id=2729666a518f4303a2934950e17ffe57)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:55.182824 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698fc205-f236-4f54-930f-9fa29f5731df admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:55.186240 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46364e1-1016-4f2d-86e1-4558b80f23d9 service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:55.193209 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cc141c-19a2-4d23-a205-ade97b7eb43d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:55.193885 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cc141c-19a2-4d23-a205-ade97b7eb43d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:55.194318 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cc141c-19a2-4d23-a205-ade97b7eb43d admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:55.194598 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cc141c-19a2-4d23-a205-ade97b7eb43d admin admin] PATH_INFO: `/v3/projects/8c7092bb4084473fa41466f1f4af3154` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:55.195841 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02cc141c-19a2-4d23-a205-ade97b7eb43d admin admin] RBAC: Authorizing `identity:delete_project(project_id=8c7092bb4084473fa41466f1f4af3154)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:55.197880 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02cc141c-19a2-4d23-a205-ade97b7eb43d admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:55.264297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-698fc205-f236-4f54-930f-9fa29f5731df admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2729666a518f4303a2934950e17ffe57', 'request_id': 'req-698fc205-f236-4f54-930f-9fa29f5731df'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:55.266591 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-02cc141c-19a2-4d23-a205-ade97b7eb43d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8c7092bb4084473fa41466f1f4af3154', 'request_id': 'req-02cc141c-19a2-4d23-a205-ade97b7eb43d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:55.273061 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-02cc141c-19a2-4d23-a205-ade97b7eb43d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8c7092bb4084473fa41466f1f4af3154', 'request_id': 'req-02cc141c-19a2-4d23-a205-ade97b7eb43d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:55.274258 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-698fc205-f236-4f54-930f-9fa29f5731df admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2729666a518f4303a2934950e17ffe57', 'request_id': 'req-698fc205-f236-4f54-930f-9fa29f5731df'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:55.281463 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-02cc141c-19a2-4d23-a205-ade97b7eb43d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8c7092bb4084473fa41466f1f4af3154', 'request_id': 'req-02cc141c-19a2-4d23-a205-ade97b7eb43d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:55.282145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-698fc205-f236-4f54-930f-9fa29f5731df admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2729666a518f4303a2934950e17ffe57', 'request_id': 'req-698fc205-f236-4f54-930f-9fa29f5731df'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:55.292534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:55.295090 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-02cc141c-19a2-4d23-a205-ade97b7eb43d admin admin] The token cache is being invalidate because project 8c7092bb4084473fa41466f1f4af3154 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:55.295393 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-02cc141c-19a2-4d23-a205-ade97b7eb43d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:55.296109 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-02cc141c-19a2-4d23-a205-ade97b7eb43d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:55.297830 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1020/2047] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:55 2022] DELETE /identity/v3/projects/8c7092bb4084473fa41466f1f4af3154 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:55.298421 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-698fc205-f236-4f54-930f-9fa29f5731df admin admin] The token cache is being invalidate because project 2729666a518f4303a2934950e17ffe57 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:55.299102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-698fc205-f236-4f54-930f-9fa29f5731df admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:55.299905 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-698fc205-f236-4f54-930f-9fa29f5731df admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:55.301991 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cc141c-19a2-4d23-a205-ade97b7eb43d admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:55.302508 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1028/2048] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:55 2022] DELETE /identity/v3/projects/2729666a518f4303a2934950e17ffe57 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:55.303502 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-02cc141c-19a2-4d23-a205-ade97b7eb43d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:55.306341 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698fc205-f236-4f54-930f-9fa29f5731df admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:55.308257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-698fc205-f236-4f54-930f-9fa29f5731df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:55.309867 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:55.379467 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cc141c-19a2-4d23-a205-ade97b7eb43d admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:55.384367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04139dc-8ab6-4b0f-913d-e49774b558e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:55.385003 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04139dc-8ab6-4b0f-913d-e49774b558e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:55.385283 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04139dc-8ab6-4b0f-913d-e49774b558e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:55.385556 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04139dc-8ab6-4b0f-913d-e49774b558e1 admin admin] PATH_INFO: `/v3/users/2ff642ed545446a5bd4ea16c56bacb5f` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:55.386721 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04139dc-8ab6-4b0f-913d-e49774b558e1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2ff642ed545446a5bd4ea16c56bacb5f)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:55.388622 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04139dc-8ab6-4b0f-913d-e49774b558e1 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:55.390674 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698fc205-f236-4f54-930f-9fa29f5731df admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:55.395313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4834fb-097a-40ce-82bf-b73e0a26dba6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:55.395932 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4834fb-097a-40ce-82bf-b73e0a26dba6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:55.396220 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4834fb-097a-40ce-82bf-b73e0a26dba6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:55.396461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4834fb-097a-40ce-82bf-b73e0a26dba6 admin admin] PATH_INFO: `/v3/users/72490b69a32649a3bea8ef3cd2002f14` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:55.397580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4834fb-097a-40ce-82bf-b73e0a26dba6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=72490b69a32649a3bea8ef3cd2002f14)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:55.399543 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4834fb-097a-40ce-82bf-b73e0a26dba6 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:55.443635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-c04139dc-8ab6-4b0f-913d-e49774b558e1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2ff642ed545446a5bd4ea16c56bacb5f'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:55.447940 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8e4834fb-097a-40ce-82bf-b73e0a26dba6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '72490b69a32649a3bea8ef3cd2002f14'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:55.451311 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-c04139dc-8ab6-4b0f-913d-e49774b558e1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2ff642ed545446a5bd4ea16c56bacb5f'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:55.453069 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-c04139dc-8ab6-4b0f-913d-e49774b558e1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2ff642ed545446a5bd4ea16c56bacb5f'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:55.455644 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8e4834fb-097a-40ce-82bf-b73e0a26dba6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '72490b69a32649a3bea8ef3cd2002f14'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:55.456323 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8e4834fb-097a-40ce-82bf-b73e0a26dba6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '72490b69a32649a3bea8ef3cd2002f14'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:55.461994 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-c04139dc-8ab6-4b0f-913d-e49774b558e1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2ff642ed545446a5bd4ea16c56bacb5f'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:55.464361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8e4834fb-097a-40ce-82bf-b73e0a26dba6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '72490b69a32649a3bea8ef3cd2002f14'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:55.474598 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-c04139dc-8ab6-4b0f-913d-e49774b558e1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2ff642ed545446a5bd4ea16c56bacb5f'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:55.480632 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1021/2049] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:55 2022] DELETE /identity/v3/users/2ff642ed545446a5bd4ea16c56bacb5f => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:55.480749 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8e4834fb-097a-40ce-82bf-b73e0a26dba6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '72490b69a32649a3bea8ef3cd2002f14'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:55.489631 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1029/2050] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:55 2022] DELETE /identity/v3/users/72490b69a32649a3bea8ef3cd2002f14 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:55.905840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04139dc-8ab6-4b0f-913d-e49774b558e1 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:55.907784 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c04139dc-8ab6-4b0f-913d-e49774b558e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:55.923891 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4834fb-097a-40ce-82bf-b73e0a26dba6 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:55.925545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-8e4834fb-097a-40ce-82bf-b73e0a26dba6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:55.997983 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04139dc-8ab6-4b0f-913d-e49774b558e1 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:56.012076 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4834fb-097a-40ce-82bf-b73e0a26dba6 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:56.017256 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800f3051-aea1-4a83-b219-9fe5f966f281 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:56.017969 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800f3051-aea1-4a83-b219-9fe5f966f281 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:56.018233 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800f3051-aea1-4a83-b219-9fe5f966f281 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:56.018507 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800f3051-aea1-4a83-b219-9fe5f966f281 admin admin] PATH_INFO: `/v3/projects/88d7c063eb474468a8bf3fbc131c3b0f` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:56.019623 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800f3051-aea1-4a83-b219-9fe5f966f281 admin admin] RBAC: Authorizing `identity:delete_project(project_id=88d7c063eb474468a8bf3fbc131c3b0f)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:56.021574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800f3051-aea1-4a83-b219-9fe5f966f281 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:56.025286 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:56.046725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:56.063647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c583706-c0c1-4f1f-86f0-9eab7edc5d7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:56.064283 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c583706-c0c1-4f1f-86f0-9eab7edc5d7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:56.064565 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c583706-c0c1-4f1f-86f0-9eab7edc5d7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:56.064918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c583706-c0c1-4f1f-86f0-9eab7edc5d7e admin admin] PATH_INFO: `/v3/projects/40ed12f0e3414ab393dd77ced315a36b` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:56.066128 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c583706-c0c1-4f1f-86f0-9eab7edc5d7e admin admin] RBAC: Authorizing `identity:delete_project(project_id=40ed12f0e3414ab393dd77ced315a36b)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:56.067947 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c583706-c0c1-4f1f-86f0-9eab7edc5d7e admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:56.074490 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-800f3051-aea1-4a83-b219-9fe5f966f281 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '88d7c063eb474468a8bf3fbc131c3b0f', 'request_id': 'req-800f3051-aea1-4a83-b219-9fe5f966f281'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:56.080449 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-800f3051-aea1-4a83-b219-9fe5f966f281 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '88d7c063eb474468a8bf3fbc131c3b0f', 'request_id': 'req-800f3051-aea1-4a83-b219-9fe5f966f281'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:56.087057 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-800f3051-aea1-4a83-b219-9fe5f966f281 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '88d7c063eb474468a8bf3fbc131c3b0f', 'request_id': 'req-800f3051-aea1-4a83-b219-9fe5f966f281'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:56.103894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-800f3051-aea1-4a83-b219-9fe5f966f281 admin admin] The token cache is being invalidate because project 88d7c063eb474468a8bf3fbc131c3b0f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:56.104186 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-800f3051-aea1-4a83-b219-9fe5f966f281 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:56.104971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-800f3051-aea1-4a83-b219-9fe5f966f281 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:56.106850 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1030/2051] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:55 2022] DELETE /identity/v3/projects/88d7c063eb474468a8bf3fbc131c3b0f => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:20:56.112109 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cee711-9553-4dc8-a068-1309d08d0dcd None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:56.112464 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cee711-9553-4dc8-a068-1309d08d0dcd None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:56.112739 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cee711-9553-4dc8-a068-1309d08d0dcd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:56.137288 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-4c583706-c0c1-4f1f-86f0-9eab7edc5d7e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '40ed12f0e3414ab393dd77ced315a36b', 'request_id': 'req-4c583706-c0c1-4f1f-86f0-9eab7edc5d7e'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:56.144390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-e9cee711-9553-4dc8-a068-1309d08d0dcd None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:56.145448 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-4c583706-c0c1-4f1f-86f0-9eab7edc5d7e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '40ed12f0e3414ab393dd77ced315a36b', 'request_id': 'req-4c583706-c0c1-4f1f-86f0-9eab7edc5d7e'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:56.145627 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-e9cee711-9553-4dc8-a068-1309d08d0dcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:56.152561 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-4c583706-c0c1-4f1f-86f0-9eab7edc5d7e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '40ed12f0e3414ab393dd77ced315a36b', 'request_id': 'req-4c583706-c0c1-4f1f-86f0-9eab7edc5d7e'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:56.173281 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-4c583706-c0c1-4f1f-86f0-9eab7edc5d7e admin admin] The token cache is being invalidate because project 40ed12f0e3414ab393dd77ced315a36b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:56.173719 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-4c583706-c0c1-4f1f-86f0-9eab7edc5d7e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:56.174636 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-4c583706-c0c1-4f1f-86f0-9eab7edc5d7e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:56.176587 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1022/2052] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:55 2022] DELETE /identity/v3/projects/40ed12f0e3414ab393dd77ced315a36b => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:56.181154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be3cb29-d8e6-4306-a64d-d47f306e2ea2 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:56.181559 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be3cb29-d8e6-4306-a64d-d47f306e2ea2 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:56.182157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be3cb29-d8e6-4306-a64d-d47f306e2ea2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:56.204310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-1be3cb29-d8e6-4306-a64d-d47f306e2ea2 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:56.205621 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-1be3cb29-d8e6-4306-a64d-d47f306e2ea2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:56.222873 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1031/2053] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:20:56.232139 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c0d539-c1ee-49ab-9533-4ec1f1327bce None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:56.232818 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c0d539-c1ee-49ab-9533-4ec1f1327bce None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:56.233920 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c0d539-c1ee-49ab-9533-4ec1f1327bce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:56.254645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-b4c0d539-c1ee-49ab-9533-4ec1f1327bce None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:56.256141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-b4c0d539-c1ee-49ab-9533-4ec1f1327bce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:56.264711 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1032/2054] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:20:56.268661 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c0d539-c1ee-49ab-9533-4ec1f1327bce None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:56.277735 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c0d539-c1ee-49ab-9533-4ec1f1327bce None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:56.289004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e342812-f854-448e-8107-d4a4658dc32d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:56.289271 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1023/2055] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:20:56.290056 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e342812-f854-448e-8107-d4a4658dc32d admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:56.290947 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e342812-f854-448e-8107-d4a4658dc32d admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:56.291480 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e342812-f854-448e-8107-d4a4658dc32d admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:56.292938 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e342812-f854-448e-8107-d4a4658dc32d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:20:56.293349 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e342812-f854-448e-8107-d4a4658dc32d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:56.296823 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e342812-f854-448e-8107-d4a4658dc32d admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:56.297673 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf10600-2e5f-4d2b-b58a-86477de87310 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:56.298300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf10600-2e5f-4d2b-b58a-86477de87310 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:56.298642 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf10600-2e5f-4d2b-b58a-86477de87310 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:56.317546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1033/2056] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:20:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:20:56.322145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e342812-f854-448e-8107-d4a4658dc32d admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:56.323499 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-3cf10600-2e5f-4d2b-b58a-86477de87310 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:56.324707 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-3cf10600-2e5f-4d2b-b58a-86477de87310 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:56.336364 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e342812-f854-448e-8107-d4a4658dc32d admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:56.340963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1024/2057] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:56.341698 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf10600-2e5f-4d2b-b58a-86477de87310 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:56.345353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9630a3db-b4fb-4b9c-80ac-cd017aec07a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:56.346109 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9630a3db-b4fb-4b9c-80ac-cd017aec07a0 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:56.346490 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9630a3db-b4fb-4b9c-80ac-cd017aec07a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:56.347060 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9630a3db-b4fb-4b9c-80ac-cd017aec07a0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:56.348275 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9630a3db-b4fb-4b9c-80ac-cd017aec07a0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:56.350811 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9630a3db-b4fb-4b9c-80ac-cd017aec07a0 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:56.352581 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf10600-2e5f-4d2b-b58a-86477de87310 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:56.359120 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23598f44-82f0-4df4-a4cb-5e1a4ace34f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:56.359963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23598f44-82f0-4df4-a4cb-5e1a4ace34f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:56.360399 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23598f44-82f0-4df4-a4cb-5e1a4ace34f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:56.360816 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23598f44-82f0-4df4-a4cb-5e1a4ace34f6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:56.362371 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23598f44-82f0-4df4-a4cb-5e1a4ace34f6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:20:56.362841 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23598f44-82f0-4df4-a4cb-5e1a4ace34f6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:56.364624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23598f44-82f0-4df4-a4cb-5e1a4ace34f6 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:56.378124 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1025/2058] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:20:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:20:56.382875 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23598f44-82f0-4df4-a4cb-5e1a4ace34f6 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:56.400881 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1034/2059] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:56 2022] POST /identity/v3/projects => generated 385 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:20:56.401014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23598f44-82f0-4df4-a4cb-5e1a4ace34f6 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:56.412162 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9630a3db-b4fb-4b9c-80ac-cd017aec07a0 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:56.482461 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:56.518937 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3afbb7-7354-46af-bc6b-06391e871cfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:56.519637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3afbb7-7354-46af-bc6b-06391e871cfa admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:56.519983 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3afbb7-7354-46af-bc6b-06391e871cfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:56.520369 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3afbb7-7354-46af-bc6b-06391e871cfa admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:56.520925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9630a3db-b4fb-4b9c-80ac-cd017aec07a0 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:56.521195 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3afbb7-7354-46af-bc6b-06391e871cfa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:56.523483 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3afbb7-7354-46af-bc6b-06391e871cfa admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:56.526305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cafe2ca-1c96-4712-ab7c-0cd5cc9deade admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:56.527057 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cafe2ca-1c96-4712-ab7c-0cd5cc9deade admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:56.527404 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cafe2ca-1c96-4712-ab7c-0cd5cc9deade admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:56.527743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cafe2ca-1c96-4712-ab7c-0cd5cc9deade admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:56.529022 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cafe2ca-1c96-4712-ab7c-0cd5cc9deade admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:56.532909 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cafe2ca-1c96-4712-ab7c-0cd5cc9deade admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:56.551542 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:56.559458 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1026/2060] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:56 2022] POST /identity/v3/projects => generated 383 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:56.563279 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3afbb7-7354-46af-bc6b-06391e871cfa admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:56.564937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1035/2061] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:56 2022] POST /identity/v3/users => generated 293 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:20:56.568874 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cafe2ca-1c96-4712-ab7c-0cd5cc9deade admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:56.581315 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cafe2ca-1c96-4712-ab7c-0cd5cc9deade admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:56.638667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3afbb7-7354-46af-bc6b-06391e871cfa admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:56.641287 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98047775-947b-4982-bf69-99855197e9f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:56.641928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98047775-947b-4982-bf69-99855197e9f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:56.642204 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98047775-947b-4982-bf69-99855197e9f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:56.642474 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98047775-947b-4982-bf69-99855197e9f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:56.643114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98047775-947b-4982-bf69-99855197e9f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:56.644168 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b3680f-433c-4db8-9dc6-86741298b35c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:56.644724 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98047775-947b-4982-bf69-99855197e9f5 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:56.644888 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b3680f-433c-4db8-9dc6-86741298b35c admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:56.645248 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b3680f-433c-4db8-9dc6-86741298b35c admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:56.645643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b3680f-433c-4db8-9dc6-86741298b35c admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:56.646391 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b3680f-433c-4db8-9dc6-86741298b35c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:56.649323 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b3680f-433c-4db8-9dc6-86741298b35c admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:56.656881 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1036/2062] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:56 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:20:56.661104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98047775-947b-4982-bf69-99855197e9f5 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:56.671062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1027/2063] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:56 2022] POST /identity/v3/users => generated 292 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:56.671224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98047775-947b-4982-bf69-99855197e9f5 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:56.675047 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b3680f-433c-4db8-9dc6-86741298b35c admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:56.675977 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c6afda-c41e-40eb-8b4a-51f59c019557 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:56.676565 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c6afda-c41e-40eb-8b4a-51f59c019557 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:56.676844 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c6afda-c41e-40eb-8b4a-51f59c019557 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:56.677118 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c6afda-c41e-40eb-8b4a-51f59c019557 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:56.677783 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c6afda-c41e-40eb-8b4a-51f59c019557 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:56.679368 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c6afda-c41e-40eb-8b4a-51f59c019557 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:56.684771 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b3680f-433c-4db8-9dc6-86741298b35c admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:56.688931 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4dd386-b667-4233-8dfe-978f714b4785 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:56.689500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4dd386-b667-4233-8dfe-978f714b4785 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:56.689817 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4dd386-b667-4233-8dfe-978f714b4785 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:56.692719 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1037/2064] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:56 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:56.692815 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4dd386-b667-4233-8dfe-978f714b4785 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:56.692815 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4dd386-b667-4233-8dfe-978f714b4785 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:56.692815 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4dd386-b667-4233-8dfe-978f714b4785 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:56.694088 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c6afda-c41e-40eb-8b4a-51f59c019557 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:56.706125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1028/2065] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:56 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:56.706272 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c6afda-c41e-40eb-8b4a-51f59c019557 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:56.707128 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4dd386-b667-4233-8dfe-978f714b4785 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:56.708821 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483cfe06-1c92-46ed-9c60-d064a61d27a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:56.709496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483cfe06-1c92-46ed-9c60-d064a61d27a0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:56.709717 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483cfe06-1c92-46ed-9c60-d064a61d27a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:56.710029 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483cfe06-1c92-46ed-9c60-d064a61d27a0 admin admin] PATH_INFO: `/v3/projects/982e342f1e1f49c7b5c7586650c5394b/users/980c438b4faa4bb38dc9c00e7358f647/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:56.716679 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4dd386-b667-4233-8dfe-978f714b4785 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:56.721967 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592ec8b7-fc76-44a6-907d-b206778a031b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:56.722556 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592ec8b7-fc76-44a6-907d-b206778a031b admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:56.723006 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592ec8b7-fc76-44a6-907d-b206778a031b admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:56.723281 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592ec8b7-fc76-44a6-907d-b206778a031b admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:56.723921 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592ec8b7-fc76-44a6-907d-b206778a031b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:56.725490 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592ec8b7-fc76-44a6-907d-b206778a031b admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:56.726417 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483cfe06-1c92-46ed-9c60-d064a61d27a0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=982e342f1e1f49c7b5c7586650c5394b, user_id=980c438b4faa4bb38dc9c00e7358f647, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:56.729115 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483cfe06-1c92-46ed-9c60-d064a61d27a0 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:56.736363 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1029/2066] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:56 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:20:56.741038 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592ec8b7-fc76-44a6-907d-b206778a031b admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:56.748634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1038/2067] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:56 2022] PUT /identity/v3/projects/982e342f1e1f49c7b5c7586650c5394b/users/980c438b4faa4bb38dc9c00e7358f647/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:56.759968 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dadaf1-4e68-461e-88a4-8992829ec73b None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:56.760253 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dadaf1-4e68-461e-88a4-8992829ec73b None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:56.760655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dadaf1-4e68-461e-88a4-8992829ec73b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:56.764956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592ec8b7-fc76-44a6-907d-b206778a031b admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:56.790941 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-20dadaf1-4e68-461e-88a4-8992829ec73b None None] MFA Rules not processed for user `980c438b4faa4bb38dc9c00e7358f647`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:56.792179 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-20dadaf1-4e68-461e-88a4-8992829ec73b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:56.836439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b5df93-0baa-4ab4-ba27-480601829c0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:56.837543 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b5df93-0baa-4ab4-ba27-480601829c0f admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:56.837931 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b5df93-0baa-4ab4-ba27-480601829c0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:56.838680 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b5df93-0baa-4ab4-ba27-480601829c0f admin admin] PATH_INFO: `/v3/projects/ffc59ae7bbce4d56aace17cdd05ecceb/users/24224702d2724959a69713c69630384c/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:56.856263 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b5df93-0baa-4ab4-ba27-480601829c0f admin admin] RBAC: Authorizing `identity:create_grant(project_id=ffc59ae7bbce4d56aace17cdd05ecceb, user_id=24224702d2724959a69713c69630384c, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:56.859676 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b5df93-0baa-4ab4-ba27-480601829c0f admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:56.863590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1039/2068] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:56 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:20:56.868520 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20dadaf1-4e68-461e-88a4-8992829ec73b None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:56.875786 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1030/2069] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:56 2022] PUT /identity/v3/projects/ffc59ae7bbce4d56aace17cdd05ecceb/users/24224702d2724959a69713c69630384c/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:56.886176 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1e22e5-3f51-45b0-aef0-b1f4f87ba367 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:56.886506 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1e22e5-3f51-45b0-aef0-b1f4f87ba367 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:56.886798 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1e22e5-3f51-45b0-aef0-b1f4f87ba367 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:56.891069 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20dadaf1-4e68-461e-88a4-8992829ec73b None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:56.923913 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-5d1e22e5-3f51-45b0-aef0-b1f4f87ba367 None None] MFA Rules not processed for user `24224702d2724959a69713c69630384c`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:56.925627 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-5d1e22e5-3f51-45b0-aef0-b1f4f87ba367 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:56.931480 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:56.971532 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:56.990239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4476dbe2-2ef4-4818-878a-df5f0f1a642a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:56.990902 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4476dbe2-2ef4-4818-878a-df5f0f1a642a admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:56.991192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4476dbe2-2ef4-4818-878a-df5f0f1a642a admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:56.991467 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4476dbe2-2ef4-4818-878a-df5f0f1a642a admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:56.998440 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4476dbe2-2ef4-4818-878a-df5f0f1a642a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:57.001939 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4476dbe2-2ef4-4818-878a-df5f0f1a642a admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:57.046245 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1040/2070] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:56 2022] POST /identity/v3/projects => generated 387 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:20:57.047590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4476dbe2-2ef4-4818-878a-df5f0f1a642a admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:57.050938 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1031/2071] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:56 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:20:57.055110 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1e22e5-3f51-45b0-aef0-b1f4f87ba367 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:57.071334 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1e22e5-3f51-45b0-aef0-b1f4f87ba367 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:57.167782 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4476dbe2-2ef4-4818-878a-df5f0f1a642a admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:57.175233 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c87d87-b2ef-4f5c-99b4-b22f89ad069d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:57.176034 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c87d87-b2ef-4f5c-99b4-b22f89ad069d admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:57.176446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a333f30-3fbc-4f14-a550-2fb5c2418419 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:57.176653 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c87d87-b2ef-4f5c-99b4-b22f89ad069d admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:57.177201 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c87d87-b2ef-4f5c-99b4-b22f89ad069d admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:57.177506 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a333f30-3fbc-4f14-a550-2fb5c2418419 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:57.177954 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a333f30-3fbc-4f14-a550-2fb5c2418419 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:57.178251 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a333f30-3fbc-4f14-a550-2fb5c2418419 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:57.178706 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c87d87-b2ef-4f5c-99b4-b22f89ad069d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:57.178887 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a333f30-3fbc-4f14-a550-2fb5c2418419 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:57.180997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c87d87-b2ef-4f5c-99b4-b22f89ad069d admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:57.181131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a333f30-3fbc-4f14-a550-2fb5c2418419 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:57.217887 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1041/2072] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:57 2022] POST /identity/v3/users => generated 300 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:57.219476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c87d87-b2ef-4f5c-99b4-b22f89ad069d admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:57.221607 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1032/2073] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:20:57 2022] POST /identity/v3/projects => generated 385 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:57.225180 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a333f30-3fbc-4f14-a550-2fb5c2418419 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:57.232981 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c87d87-b2ef-4f5c-99b4-b22f89ad069d admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:57.324313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad0025a-8102-40de-8d4c-be5a74acf5d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:57.325091 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad0025a-8102-40de-8d4c-be5a74acf5d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:57.325376 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad0025a-8102-40de-8d4c-be5a74acf5d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:57.325650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad0025a-8102-40de-8d4c-be5a74acf5d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:57.326331 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad0025a-8102-40de-8d4c-be5a74acf5d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:57.328094 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad0025a-8102-40de-8d4c-be5a74acf5d9 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:57.341502 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a333f30-3fbc-4f14-a550-2fb5c2418419 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:57.346049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1042/2074] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:57 2022] GET /identity/v3/roles => generated 1420 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:20:57.350211 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad0025a-8102-40de-8d4c-be5a74acf5d9 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:57.351595 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ba81df-7a62-4c64-877c-2be2ad4dada2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:57.352376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ba81df-7a62-4c64-877c-2be2ad4dada2 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:57.352764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ba81df-7a62-4c64-877c-2be2ad4dada2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:57.353368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ba81df-7a62-4c64-877c-2be2ad4dada2 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:57.354336 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ba81df-7a62-4c64-877c-2be2ad4dada2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:57.356460 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ba81df-7a62-4c64-877c-2be2ad4dada2 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:57.379112 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad0025a-8102-40de-8d4c-be5a74acf5d9 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:57.385472 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f254ff-4864-414f-9082-974ef7e08b46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:57.386229 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f254ff-4864-414f-9082-974ef7e08b46 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:57.396130 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1033/2075] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:20:57 2022] POST /identity/v3/users => generated 299 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:20:57.399168 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f254ff-4864-414f-9082-974ef7e08b46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:57.399538 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f254ff-4864-414f-9082-974ef7e08b46 admin admin] PATH_INFO: `/v3/projects/769231fa56cf43228509a0a783d7d28d/users/8612a7549d514df1b83546e6687619bd/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:57.401006 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ba81df-7a62-4c64-877c-2be2ad4dada2 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:57.428704 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ba81df-7a62-4c64-877c-2be2ad4dada2 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:57.440140 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8492dd-e0a0-45d2-b6e5-6a685288f577 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:57.440782 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8492dd-e0a0-45d2-b6e5-6a685288f577 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:57.441067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8492dd-e0a0-45d2-b6e5-6a685288f577 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:57.441342 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8492dd-e0a0-45d2-b6e5-6a685288f577 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:57.441536 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f254ff-4864-414f-9082-974ef7e08b46 admin admin] RBAC: Authorizing `identity:create_grant(project_id=769231fa56cf43228509a0a783d7d28d, user_id=8612a7549d514df1b83546e6687619bd, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:57.442040 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d8492dd-e0a0-45d2-b6e5-6a685288f577 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:57.443784 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d8492dd-e0a0-45d2-b6e5-6a685288f577 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:57.446270 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f254ff-4864-414f-9082-974ef7e08b46 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:57.460120 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1034/2076] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:57 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:20:57.465930 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8492dd-e0a0-45d2-b6e5-6a685288f577 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:57.472093 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1043/2077] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:57 2022] PUT /identity/v3/projects/769231fa56cf43228509a0a783d7d28d/users/8612a7549d514df1b83546e6687619bd/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:57.476147 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f254ff-4864-414f-9082-974ef7e08b46 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:57.490115 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8492dd-e0a0-45d2-b6e5-6a685288f577 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:57.499914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:57.529054 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:57.610856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f254ff-4864-414f-9082-974ef7e08b46 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:57.626949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04440c0a-594a-42d4-90aa-9861ca7d39d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:57.627746 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04440c0a-594a-42d4-90aa-9861ca7d39d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:57.640361 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91923927-7045-4ac1-a4c0-448b575bb224 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:57.640883 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91923927-7045-4ac1-a4c0-448b575bb224 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:57.641171 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91923927-7045-4ac1-a4c0-448b575bb224 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:57.641430 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04440c0a-594a-42d4-90aa-9861ca7d39d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:57.641643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91923927-7045-4ac1-a4c0-448b575bb224 admin admin] PATH_INFO: `/v3/projects/91f011314af940eda4e4fdb66a3c8226/users/525171b955ed4b7480fcc720d0cf4d5c/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:57.642322 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04440c0a-594a-42d4-90aa-9861ca7d39d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:57.644283 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04440c0a-594a-42d4-90aa-9861ca7d39d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:57.649412 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04440c0a-594a-42d4-90aa-9861ca7d39d1 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:57.679611 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1044/2078] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:57 2022] GET /identity/v3/roles => generated 1420 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:20:57.683167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91923927-7045-4ac1-a4c0-448b575bb224 admin admin] RBAC: Authorizing `identity:create_grant(project_id=91f011314af940eda4e4fdb66a3c8226, user_id=525171b955ed4b7480fcc720d0cf4d5c, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:57.684580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04440c0a-594a-42d4-90aa-9861ca7d39d1 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:57.686390 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91923927-7045-4ac1-a4c0-448b575bb224 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:57.701728 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04440c0a-594a-42d4-90aa-9861ca7d39d1 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:57.705924 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1035/2079] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:57 2022] PUT /identity/v3/projects/91f011314af940eda4e4fdb66a3c8226/users/525171b955ed4b7480fcc720d0cf4d5c/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:57.714270 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91923927-7045-4ac1-a4c0-448b575bb224 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:57.782734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86eebf4-e815-4310-bed5-c592e0bc635a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:57.783415 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86eebf4-e815-4310-bed5-c592e0bc635a admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:57.783691 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86eebf4-e815-4310-bed5-c592e0bc635a admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:57.783959 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86eebf4-e815-4310-bed5-c592e0bc635a admin admin] PATH_INFO: `/v3/projects/769231fa56cf43228509a0a783d7d28d/users/8612a7549d514df1b83546e6687619bd/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:57.786382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c86eebf4-e815-4310-bed5-c592e0bc635a admin admin] RBAC: Authorizing `identity:create_grant(project_id=769231fa56cf43228509a0a783d7d28d, user_id=8612a7549d514df1b83546e6687619bd, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:57.789412 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c86eebf4-e815-4310-bed5-c592e0bc635a admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:57.797179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91923927-7045-4ac1-a4c0-448b575bb224 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:57.809827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1045/2080] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:57 2022] PUT /identity/v3/projects/769231fa56cf43228509a0a783d7d28d/users/8612a7549d514df1b83546e6687619bd/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:20:57.816117 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c1995d-adc3-450c-b537-c99655ac5a22 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:57.816579 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c1995d-adc3-450c-b537-c99655ac5a22 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:57.817313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c1995d-adc3-450c-b537-c99655ac5a22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:57.865852 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-00c1995d-adc3-450c-b537-c99655ac5a22 None None] MFA Rules not processed for user `8612a7549d514df1b83546e6687619bd`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:57.867097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-00c1995d-adc3-450c-b537-c99655ac5a22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:57.887857 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64a18ce-4311-46c4-807d-2a8f1b785811 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:57.889009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64a18ce-4311-46c4-807d-2a8f1b785811 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:57.890199 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64a18ce-4311-46c4-807d-2a8f1b785811 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:57.890665 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64a18ce-4311-46c4-807d-2a8f1b785811 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:57.891815 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64a18ce-4311-46c4-807d-2a8f1b785811 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:57.894100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64a18ce-4311-46c4-807d-2a8f1b785811 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:57.906900 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1036/2081] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:20:57 2022] GET /identity/v3/roles => generated 1420 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:20:57.912641 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64a18ce-4311-46c4-807d-2a8f1b785811 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:57.928658 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64a18ce-4311-46c4-807d-2a8f1b785811 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:57.933540 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b8bf31-ee25-4a5d-9708-450dc9a87a69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:57.934410 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b8bf31-ee25-4a5d-9708-450dc9a87a69 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:57.935089 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b8bf31-ee25-4a5d-9708-450dc9a87a69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:57.935257 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b8bf31-ee25-4a5d-9708-450dc9a87a69 admin admin] PATH_INFO: `/v3/projects/91f011314af940eda4e4fdb66a3c8226/users/525171b955ed4b7480fcc720d0cf4d5c/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:57.937670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b8bf31-ee25-4a5d-9708-450dc9a87a69 admin admin] RBAC: Authorizing `identity:create_grant(project_id=91f011314af940eda4e4fdb66a3c8226, user_id=525171b955ed4b7480fcc720d0cf4d5c, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:57.940481 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b8bf31-ee25-4a5d-9708-450dc9a87a69 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:57.963249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1037/2082] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:20:57 2022] PUT /identity/v3/projects/91f011314af940eda4e4fdb66a3c8226/users/525171b955ed4b7480fcc720d0cf4d5c/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:57.974680 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a8ae3c-ac07-4e22-af54-50861a12dc0f None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:57.975072 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a8ae3c-ac07-4e22-af54-50861a12dc0f None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:57.975403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a8ae3c-ac07-4e22-af54-50861a12dc0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:57.987776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1046/2083] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:57 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:58.003403 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c1995d-adc3-450c-b537-c99655ac5a22 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:58.005919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-00c1995d-adc3-450c-b537-c99655ac5a22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:58.010683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-c2a8ae3c-ac07-4e22-af54-50861a12dc0f None None] MFA Rules not processed for user `525171b955ed4b7480fcc720d0cf4d5c`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:58.012535 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c2a8ae3c-ac07-4e22-af54-50861a12dc0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:58.027016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:58.076537 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:58.083301 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c1995d-adc3-450c-b537-c99655ac5a22 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:58.100842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1038/2084] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:57 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:58.112811 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a8ae3c-ac07-4e22-af54-50861a12dc0f None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:58.114315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c2a8ae3c-ac07-4e22-af54-50861a12dc0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:58.149293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a34e25-97cf-4b34-9358-f4cd75416c3a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:58.150007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a34e25-97cf-4b34-9358-f4cd75416c3a service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:58.150289 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a34e25-97cf-4b34-9358-f4cd75416c3a service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:58.150653 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a34e25-97cf-4b34-9358-f4cd75416c3a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:58.187854 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a8ae3c-ac07-4e22-af54-50861a12dc0f None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:58.192093 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86f287d-86af-4cf1-9d24-2daf4f16e5cb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:58.192712 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86f287d-86af-4cf1-9d24-2daf4f16e5cb service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:58.192989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86f287d-86af-4cf1-9d24-2daf4f16e5cb service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:58.193442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86f287d-86af-4cf1-9d24-2daf4f16e5cb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:58.233194 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a34e25-97cf-4b34-9358-f4cd75416c3a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:58.234783 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a34e25-97cf-4b34-9358-f4cd75416c3a service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:58.246212 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1047/2085] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:58 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 243 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:58.254574 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86f287d-86af-4cf1-9d24-2daf4f16e5cb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:58.256040 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86f287d-86af-4cf1-9d24-2daf4f16e5cb service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:58.266389 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1039/2086] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:58 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:20:58.643393 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a34e25-97cf-4b34-9358-f4cd75416c3a service cinder] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:58.645374 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-94a34e25-97cf-4b34-9358-f4cd75416c3a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:58.753956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a34e25-97cf-4b34-9358-f4cd75416c3a service cinder] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:58.759075 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd4a832-f9a3-491c-a11f-85674b569e24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:58.759858 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd4a832-f9a3-491c-a11f-85674b569e24 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:58.760222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd4a832-f9a3-491c-a11f-85674b569e24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:58.760470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd4a832-f9a3-491c-a11f-85674b569e24 admin admin] PATH_INFO: `/v3/users/fd3c3da76ddf4283a1bf5bcb8b606322` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:58.762854 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd4a832-f9a3-491c-a11f-85674b569e24 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fd3c3da76ddf4283a1bf5bcb8b606322)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:58.765292 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd4a832-f9a3-491c-a11f-85674b569e24 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:58.826687 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-7bd4a832-f9a3-491c-a11f-85674b569e24 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fd3c3da76ddf4283a1bf5bcb8b606322'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:58.836109 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-7bd4a832-f9a3-491c-a11f-85674b569e24 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fd3c3da76ddf4283a1bf5bcb8b606322'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:58.836836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-7bd4a832-f9a3-491c-a11f-85674b569e24 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fd3c3da76ddf4283a1bf5bcb8b606322'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:58.843994 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-7bd4a832-f9a3-491c-a11f-85674b569e24 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fd3c3da76ddf4283a1bf5bcb8b606322'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:58.846155 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:58.862411 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-7bd4a832-f9a3-491c-a11f-85674b569e24 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fd3c3da76ddf4283a1bf5bcb8b606322'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:58.873332 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1048/2087] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:58 2022] DELETE /identity/v3/users/fd3c3da76ddf4283a1bf5bcb8b606322 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:58.878155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0075b38-856a-462b-ab55-f80dc60ab549 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:58.878602 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0075b38-856a-462b-ab55-f80dc60ab549 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:58.878901 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0075b38-856a-462b-ab55-f80dc60ab549 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:58.906780 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-b0075b38-856a-462b-ab55-f80dc60ab549 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:58.908053 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-b0075b38-856a-462b-ab55-f80dc60ab549 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:58.981728 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd4a832-f9a3-491c-a11f-85674b569e24 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:58.985016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-7bd4a832-f9a3-491c-a11f-85674b569e24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:59.015875 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1040/2088] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:20:59.039395 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0075b38-856a-462b-ab55-f80dc60ab549 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:59.041017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-b0075b38-856a-462b-ab55-f80dc60ab549 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:59.049718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:59.082467 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd4a832-f9a3-491c-a11f-85674b569e24 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:59.090208 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6823bf04-92c2-4f60-8351-c233158ddecb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:59.091361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6823bf04-92c2-4f60-8351-c233158ddecb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:59.091822 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6823bf04-92c2-4f60-8351-c233158ddecb admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:59.092139 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6823bf04-92c2-4f60-8351-c233158ddecb admin admin] PATH_INFO: `/v3/users/980c438b4faa4bb38dc9c00e7358f647` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:59.094036 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6823bf04-92c2-4f60-8351-c233158ddecb admin admin] RBAC: Authorizing `identity:delete_user(user_id=980c438b4faa4bb38dc9c00e7358f647)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:59.096289 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6823bf04-92c2-4f60-8351-c233158ddecb admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:59.135155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0075b38-856a-462b-ab55-f80dc60ab549 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:59.141734 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c2a2a1-c4d1-46f5-a949-7d86317617a9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:59.142572 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c2a2a1-c4d1-46f5-a949-7d86317617a9 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:59.142894 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c2a2a1-c4d1-46f5-a949-7d86317617a9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:59.143280 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c2a2a1-c4d1-46f5-a949-7d86317617a9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:59.156008 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c2a2a1-c4d1-46f5-a949-7d86317617a9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:59.158252 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c2a2a1-c4d1-46f5-a949-7d86317617a9 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:59.163237 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6823bf04-92c2-4f60-8351-c233158ddecb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '980c438b4faa4bb38dc9c00e7358f647'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:59.169713 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1041/2089] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:20:59.186289 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6823bf04-92c2-4f60-8351-c233158ddecb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '980c438b4faa4bb38dc9c00e7358f647'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:59.186289 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6823bf04-92c2-4f60-8351-c233158ddecb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '980c438b4faa4bb38dc9c00e7358f647'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:59.197539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6823bf04-92c2-4f60-8351-c233158ddecb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '980c438b4faa4bb38dc9c00e7358f647'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:59.216340 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6823bf04-92c2-4f60-8351-c233158ddecb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '980c438b4faa4bb38dc9c00e7358f647'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:59.225659 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1049/2090] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:58 2022] DELETE /identity/v3/users/980c438b4faa4bb38dc9c00e7358f647 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:59.230756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78a77e5-9ede-4611-8918-c09e42214fce None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:59.231052 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78a77e5-9ede-4611-8918-c09e42214fce None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:59.231387 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78a77e5-9ede-4611-8918-c09e42214fce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:59.255573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-f78a77e5-9ede-4611-8918-c09e42214fce None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:20:59.257592 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-f78a77e5-9ede-4611-8918-c09e42214fce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:59.336799 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1042/2091] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:20:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:20:59.355941 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6823bf04-92c2-4f60-8351-c233158ddecb admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:59.358078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6823bf04-92c2-4f60-8351-c233158ddecb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:59.480217 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6823bf04-92c2-4f60-8351-c233158ddecb admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:59.485041 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9d3e6e-cd70-4817-9db7-c9ebe81a0868 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:59.485880 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9d3e6e-cd70-4817-9db7-c9ebe81a0868 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:59.486199 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9d3e6e-cd70-4817-9db7-c9ebe81a0868 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:59.486545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9d3e6e-cd70-4817-9db7-c9ebe81a0868 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:59.495219 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:INSERT=2,keystone:DELETE=13,keystone:SELECT=85 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:59.497968 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9d3e6e-cd70-4817-9db7-c9ebe81a0868 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:59.499803 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9d3e6e-cd70-4817-9db7-c9ebe81a0868 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:59.514178 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1050/2092] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:20:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:20:59.540044 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78a77e5-9ede-4611-8918-c09e42214fce None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:59.542316 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-f78a77e5-9ede-4611-8918-c09e42214fce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:59.559419 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78a77e5-9ede-4611-8918-c09e42214fce None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:59.564804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089c9cb4-a75d-4faf-89c5-6c29b338cd1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:59.565461 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089c9cb4-a75d-4faf-89c5-6c29b338cd1f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:59.565842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089c9cb4-a75d-4faf-89c5-6c29b338cd1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:59.566125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089c9cb4-a75d-4faf-89c5-6c29b338cd1f admin admin] PATH_INFO: `/v3/projects/3690b5efde2e4646ae992f5599eadb2f` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:59.567996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089c9cb4-a75d-4faf-89c5-6c29b338cd1f admin admin] RBAC: Authorizing `identity:delete_project(project_id=3690b5efde2e4646ae992f5599eadb2f)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:59.570788 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089c9cb4-a75d-4faf-89c5-6c29b338cd1f admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:59.619766 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-089c9cb4-a75d-4faf-89c5-6c29b338cd1f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3690b5efde2e4646ae992f5599eadb2f', 'request_id': 'req-089c9cb4-a75d-4faf-89c5-6c29b338cd1f'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:59.628752 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-089c9cb4-a75d-4faf-89c5-6c29b338cd1f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3690b5efde2e4646ae992f5599eadb2f', 'request_id': 'req-089c9cb4-a75d-4faf-89c5-6c29b338cd1f'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:59.635583 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-089c9cb4-a75d-4faf-89c5-6c29b338cd1f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3690b5efde2e4646ae992f5599eadb2f', 'request_id': 'req-089c9cb4-a75d-4faf-89c5-6c29b338cd1f'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:59.649039 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-089c9cb4-a75d-4faf-89c5-6c29b338cd1f admin admin] The token cache is being invalidate because project 3690b5efde2e4646ae992f5599eadb2f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:59.649330 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-089c9cb4-a75d-4faf-89c5-6c29b338cd1f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:59.650073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-089c9cb4-a75d-4faf-89c5-6c29b338cd1f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:59.651808 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1043/2093] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:59 2022] DELETE /identity/v3/projects/3690b5efde2e4646ae992f5599eadb2f => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:59.656794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9d3e6e-cd70-4817-9db7-c9ebe81a0868 service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:59.666625 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9d3e6e-cd70-4817-9db7-c9ebe81a0868 service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:59.748604 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5c5d3b-f3fe-4361-94cd-78e6a7e30933 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:59.749376 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5c5d3b-f3fe-4361-94cd-78e6a7e30933 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:59.749825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5c5d3b-f3fe-4361-94cd-78e6a7e30933 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:59.750171 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5c5d3b-f3fe-4361-94cd-78e6a7e30933 admin admin] PATH_INFO: `/v3/users/24e32354b9904dca9cc7898c87a03d6f` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:59.751388 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5c5d3b-f3fe-4361-94cd-78e6a7e30933 admin admin] RBAC: Authorizing `identity:delete_user(user_id=24e32354b9904dca9cc7898c87a03d6f)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:59.753345 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5c5d3b-f3fe-4361-94cd-78e6a7e30933 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:59.823352 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089c9cb4-a75d-4faf-89c5-6c29b338cd1f admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:59.824044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-cc5c5d3b-f3fe-4361-94cd-78e6a7e30933 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '24e32354b9904dca9cc7898c87a03d6f'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:59.824861 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-089c9cb4-a75d-4faf-89c5-6c29b338cd1f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:20:59.837982 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-cc5c5d3b-f3fe-4361-94cd-78e6a7e30933 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '24e32354b9904dca9cc7898c87a03d6f'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:59.839225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-cc5c5d3b-f3fe-4361-94cd-78e6a7e30933 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '24e32354b9904dca9cc7898c87a03d6f'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:59.853997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-cc5c5d3b-f3fe-4361-94cd-78e6a7e30933 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '24e32354b9904dca9cc7898c87a03d6f'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:59.854116 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089c9cb4-a75d-4faf-89c5-6c29b338cd1f admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:20:59.861908 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616acaaa-d469-4f25-8ad9-1950b9d3d11c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:20:59.863225 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616acaaa-d469-4f25-8ad9-1950b9d3d11c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:20:59.863647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616acaaa-d469-4f25-8ad9-1950b9d3d11c admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:20:59.863839 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616acaaa-d469-4f25-8ad9-1950b9d3d11c admin admin] PATH_INFO: `/v3/projects/982e342f1e1f49c7b5c7586650c5394b` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:20:59.864250 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-cc5c5d3b-f3fe-4361-94cd-78e6a7e30933 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '24e32354b9904dca9cc7898c87a03d6f'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:59.865215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616acaaa-d469-4f25-8ad9-1950b9d3d11c admin admin] RBAC: Authorizing `identity:delete_project(project_id=982e342f1e1f49c7b5c7586650c5394b)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:20:59.867963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616acaaa-d469-4f25-8ad9-1950b9d3d11c admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:20:59.872682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1051/2094] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:59 2022] DELETE /identity/v3/users/24e32354b9904dca9cc7898c87a03d6f => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:59.899670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:20:59.934856 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-616acaaa-d469-4f25-8ad9-1950b9d3d11c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '982e342f1e1f49c7b5c7586650c5394b', 'request_id': 'req-616acaaa-d469-4f25-8ad9-1950b9d3d11c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:59.942594 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-616acaaa-d469-4f25-8ad9-1950b9d3d11c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '982e342f1e1f49c7b5c7586650c5394b', 'request_id': 'req-616acaaa-d469-4f25-8ad9-1950b9d3d11c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:59.949605 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-616acaaa-d469-4f25-8ad9-1950b9d3d11c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '982e342f1e1f49c7b5c7586650c5394b', 'request_id': 'req-616acaaa-d469-4f25-8ad9-1950b9d3d11c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:59.966773 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-616acaaa-d469-4f25-8ad9-1950b9d3d11c admin admin] The token cache is being invalidate because project 982e342f1e1f49c7b5c7586650c5394b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:20:59.967092 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-616acaaa-d469-4f25-8ad9-1950b9d3d11c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:59.967789 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-616acaaa-d469-4f25-8ad9-1950b9d3d11c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:20:59.969775 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1044/2095] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:20:59 2022] DELETE /identity/v3/projects/982e342f1e1f49c7b5c7586650c5394b => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:20:59.973658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5c5d3b-f3fe-4361-94cd-78e6a7e30933 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:20:59.976288 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-cc5c5d3b-f3fe-4361-94cd-78e6a7e30933 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:00.081528 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5c5d3b-f3fe-4361-94cd-78e6a7e30933 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:00.106107 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:00.183577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ca1fa5-ed93-436a-b3bf-7c84f055fb59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:00.184461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ca1fa5-ed93-436a-b3bf-7c84f055fb59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:00.184740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ca1fa5-ed93-436a-b3bf-7c84f055fb59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:00.185008 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ca1fa5-ed93-436a-b3bf-7c84f055fb59 admin admin] PATH_INFO: `/v3/users/8612a7549d514df1b83546e6687619bd` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:00.187472 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ca1fa5-ed93-436a-b3bf-7c84f055fb59 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8612a7549d514df1b83546e6687619bd)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:00.191262 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ca1fa5-ed93-436a-b3bf-7c84f055fb59 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:00.266783 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616acaaa-d469-4f25-8ad9-1950b9d3d11c admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:00.268556 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-616acaaa-d469-4f25-8ad9-1950b9d3d11c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:00.285871 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-52ca1fa5-ed93-436a-b3bf-7c84f055fb59 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8612a7549d514df1b83546e6687619bd'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:00.294546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-52ca1fa5-ed93-436a-b3bf-7c84f055fb59 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8612a7549d514df1b83546e6687619bd'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:00.295309 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-52ca1fa5-ed93-436a-b3bf-7c84f055fb59 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8612a7549d514df1b83546e6687619bd'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:00.303567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616acaaa-d469-4f25-8ad9-1950b9d3d11c admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:00.305062 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-52ca1fa5-ed93-436a-b3bf-7c84f055fb59 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8612a7549d514df1b83546e6687619bd'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:00.310743 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa8d74f-a651-47cd-93a2-8af891c47503 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:00.311968 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa8d74f-a651-47cd-93a2-8af891c47503 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:00.312261 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa8d74f-a651-47cd-93a2-8af891c47503 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:00.312536 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa8d74f-a651-47cd-93a2-8af891c47503 admin admin] PATH_INFO: `/v3/projects/0716bfc7dc944ea0804a8631b5b066e0` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:00.325090 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa8d74f-a651-47cd-93a2-8af891c47503 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0716bfc7dc944ea0804a8631b5b066e0)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:00.325502 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa8d74f-a651-47cd-93a2-8af891c47503 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:00.334293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-52ca1fa5-ed93-436a-b3bf-7c84f055fb59 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8612a7549d514df1b83546e6687619bd'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:00.340912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1052/2096] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:20:59 2022] DELETE /identity/v3/users/8612a7549d514df1b83546e6687619bd => generated 0 bytes in 368 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:00.398827 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0fa8d74f-a651-47cd-93a2-8af891c47503 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0716bfc7dc944ea0804a8631b5b066e0', 'request_id': 'req-0fa8d74f-a651-47cd-93a2-8af891c47503'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:00.412220 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0fa8d74f-a651-47cd-93a2-8af891c47503 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0716bfc7dc944ea0804a8631b5b066e0', 'request_id': 'req-0fa8d74f-a651-47cd-93a2-8af891c47503'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:00.420378 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0fa8d74f-a651-47cd-93a2-8af891c47503 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0716bfc7dc944ea0804a8631b5b066e0', 'request_id': 'req-0fa8d74f-a651-47cd-93a2-8af891c47503'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:00.437966 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0fa8d74f-a651-47cd-93a2-8af891c47503 admin admin] The token cache is being invalidate because project 0716bfc7dc944ea0804a8631b5b066e0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:21:00.438513 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0fa8d74f-a651-47cd-93a2-8af891c47503 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:00.443406 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-0fa8d74f-a651-47cd-93a2-8af891c47503 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:00.445168 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1045/2097] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:21:00 2022] DELETE /identity/v3/projects/0716bfc7dc944ea0804a8631b5b066e0 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:21:00.449718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ca1fa5-ed93-436a-b3bf-7c84f055fb59 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:00.452626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-52ca1fa5-ed93-436a-b3bf-7c84f055fb59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:00.565025 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ca1fa5-ed93-436a-b3bf-7c84f055fb59 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:00.619393 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:00.641650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7397eb-75fa-42bf-b8f4-a639192203d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:00.641650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7397eb-75fa-42bf-b8f4-a639192203d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:00.641650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7397eb-75fa-42bf-b8f4-a639192203d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:00.641650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7397eb-75fa-42bf-b8f4-a639192203d6 admin admin] PATH_INFO: `/v3/users/7c091688c92e4a5683e3ac17944f03fa` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:00.642739 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7397eb-75fa-42bf-b8f4-a639192203d6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7c091688c92e4a5683e3ac17944f03fa)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:00.644839 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7397eb-75fa-42bf-b8f4-a639192203d6 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:00.709157 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6e7397eb-75fa-42bf-b8f4-a639192203d6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7c091688c92e4a5683e3ac17944f03fa'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:00.716629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6e7397eb-75fa-42bf-b8f4-a639192203d6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7c091688c92e4a5683e3ac17944f03fa'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:00.717345 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6e7397eb-75fa-42bf-b8f4-a639192203d6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7c091688c92e4a5683e3ac17944f03fa'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:00.731362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6e7397eb-75fa-42bf-b8f4-a639192203d6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7c091688c92e4a5683e3ac17944f03fa'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:00.749293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6e7397eb-75fa-42bf-b8f4-a639192203d6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7c091688c92e4a5683e3ac17944f03fa'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:00.757578 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1053/2098] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:21:00 2022] DELETE /identity/v3/users/7c091688c92e4a5683e3ac17944f03fa => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:00.996924 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa8d74f-a651-47cd-93a2-8af891c47503 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:00.998502 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0fa8d74f-a651-47cd-93a2-8af891c47503 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:01.017903 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa8d74f-a651-47cd-93a2-8af891c47503 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:01.091878 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e654eb0-a59b-40d0-8ce3-fdbec10b4942 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:01.095344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e654eb0-a59b-40d0-8ce3-fdbec10b4942 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:01.095638 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e654eb0-a59b-40d0-8ce3-fdbec10b4942 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:01.095918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e654eb0-a59b-40d0-8ce3-fdbec10b4942 admin admin] PATH_INFO: `/v3/projects/769231fa56cf43228509a0a783d7d28d` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:01.097362 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e654eb0-a59b-40d0-8ce3-fdbec10b4942 admin admin] RBAC: Authorizing `identity:delete_project(project_id=769231fa56cf43228509a0a783d7d28d)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:01.099382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e654eb0-a59b-40d0-8ce3-fdbec10b4942 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:01.164601 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:01.182056 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-5e654eb0-a59b-40d0-8ce3-fdbec10b4942 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '769231fa56cf43228509a0a783d7d28d', 'request_id': 'req-5e654eb0-a59b-40d0-8ce3-fdbec10b4942'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:01.191821 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-5e654eb0-a59b-40d0-8ce3-fdbec10b4942 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '769231fa56cf43228509a0a783d7d28d', 'request_id': 'req-5e654eb0-a59b-40d0-8ce3-fdbec10b4942'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:01.199182 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-5e654eb0-a59b-40d0-8ce3-fdbec10b4942 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '769231fa56cf43228509a0a783d7d28d', 'request_id': 'req-5e654eb0-a59b-40d0-8ce3-fdbec10b4942'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:01.232530 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-5e654eb0-a59b-40d0-8ce3-fdbec10b4942 admin admin] The token cache is being invalidate because project 769231fa56cf43228509a0a783d7d28d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:21:01.233451 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-5e654eb0-a59b-40d0-8ce3-fdbec10b4942 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:01.234366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-5e654eb0-a59b-40d0-8ce3-fdbec10b4942 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:01.236442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1046/2099] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:21:00 2022] DELETE /identity/v3/projects/769231fa56cf43228509a0a783d7d28d => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:01.248254 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a5807e-9d95-4653-9a7b-df7d011b7461 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:01.248254 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a5807e-9d95-4653-9a7b-df7d011b7461 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:01.248457 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a5807e-9d95-4653-9a7b-df7d011b7461 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:01.269890 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-b3a5807e-9d95-4653-9a7b-df7d011b7461 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:01.271155 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-b3a5807e-9d95-4653-9a7b-df7d011b7461 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:01.358716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1054/2100] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:01.361180 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794a3185-c46f-4f22-afb8-a7178f251657 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:01.361549 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794a3185-c46f-4f22-afb8-a7178f251657 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:01.364522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794a3185-c46f-4f22-afb8-a7178f251657 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:01.395200 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-794a3185-c46f-4f22-afb8-a7178f251657 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:01.396571 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-794a3185-c46f-4f22-afb8-a7178f251657 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:01.402439 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a5807e-9d95-4653-9a7b-df7d011b7461 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:01.405867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-b3a5807e-9d95-4653-9a7b-df7d011b7461 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:01.413072 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1047/2101] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:01.422981 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794a3185-c46f-4f22-afb8-a7178f251657 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:01.436138 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794a3185-c46f-4f22-afb8-a7178f251657 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:01.444227 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a57bde-17cb-4649-b5de-2f0b313824ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:01.445387 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a57bde-17cb-4649-b5de-2f0b313824ee admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:01.445791 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a57bde-17cb-4649-b5de-2f0b313824ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:01.446269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a57bde-17cb-4649-b5de-2f0b313824ee admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:01.446499 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a5807e-9d95-4653-9a7b-df7d011b7461 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:01.447931 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a57bde-17cb-4649-b5de-2f0b313824ee admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:21:01.448523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a57bde-17cb-4649-b5de-2f0b313824ee admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:01.451078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a57bde-17cb-4649-b5de-2f0b313824ee admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:01.462251 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0422ebc6-6086-4a56-a3a1-c8481fbe018d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:01.463148 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0422ebc6-6086-4a56-a3a1-c8481fbe018d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:01.464082 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0422ebc6-6086-4a56-a3a1-c8481fbe018d admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:01.464370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0422ebc6-6086-4a56-a3a1-c8481fbe018d admin admin] PATH_INFO: `/v3/projects/a0b62143bb2c44e39402a5483e0c75d2` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:01.466864 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0422ebc6-6086-4a56-a3a1-c8481fbe018d admin admin] RBAC: Authorizing `identity:delete_project(project_id=a0b62143bb2c44e39402a5483e0c75d2)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:01.468020 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1048/2102] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:21:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:21:01.475210 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0422ebc6-6086-4a56-a3a1-c8481fbe018d admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:01.478668 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a57bde-17cb-4649-b5de-2f0b313824ee admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:01.491718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a57bde-17cb-4649-b5de-2f0b313824ee admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:01.498366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc9a0a0-43c6-4380-8959-3a3a8c9d61a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:01.500295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc9a0a0-43c6-4380-8959-3a3a8c9d61a7 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:01.500585 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc9a0a0-43c6-4380-8959-3a3a8c9d61a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:01.500866 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc9a0a0-43c6-4380-8959-3a3a8c9d61a7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:01.502485 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc9a0a0-43c6-4380-8959-3a3a8c9d61a7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:01.506886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc9a0a0-43c6-4380-8959-3a3a8c9d61a7 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:01.549100 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0422ebc6-6086-4a56-a3a1-c8481fbe018d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a0b62143bb2c44e39402a5483e0c75d2', 'request_id': 'req-0422ebc6-6086-4a56-a3a1-c8481fbe018d'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:01.555104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1049/2103] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:21:01 2022] POST /identity/v3/projects => generated 385 bytes in 83 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:21:01.559743 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc9a0a0-43c6-4380-8959-3a3a8c9d61a7 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:01.560424 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0422ebc6-6086-4a56-a3a1-c8481fbe018d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a0b62143bb2c44e39402a5483e0c75d2', 'request_id': 'req-0422ebc6-6086-4a56-a3a1-c8481fbe018d'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:01.570890 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0422ebc6-6086-4a56-a3a1-c8481fbe018d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a0b62143bb2c44e39402a5483e0c75d2', 'request_id': 'req-0422ebc6-6086-4a56-a3a1-c8481fbe018d'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:01.573108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:01.591061 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0422ebc6-6086-4a56-a3a1-c8481fbe018d admin admin] The token cache is being invalidate because project a0b62143bb2c44e39402a5483e0c75d2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:21:01.591695 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0422ebc6-6086-4a56-a3a1-c8481fbe018d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:01.592947 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0422ebc6-6086-4a56-a3a1-c8481fbe018d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:01.594912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1055/2104] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:21:01 2022] DELETE /identity/v3/projects/a0b62143bb2c44e39402a5483e0c75d2 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:01.601389 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620deb01-9977-4168-86bc-8d43480129eb None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:01.601846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620deb01-9977-4168-86bc-8d43480129eb None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:01.602099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620deb01-9977-4168-86bc-8d43480129eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:01.630737 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-620deb01-9977-4168-86bc-8d43480129eb None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:01.632090 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-620deb01-9977-4168-86bc-8d43480129eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:01.678236 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc9a0a0-43c6-4380-8959-3a3a8c9d61a7 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:01.685310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25315b7-7180-4801-9244-f7f0bdff0cdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:01.690796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25315b7-7180-4801-9244-f7f0bdff0cdd admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:01.690796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25315b7-7180-4801-9244-f7f0bdff0cdd admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:01.690796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25315b7-7180-4801-9244-f7f0bdff0cdd admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:01.690796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25315b7-7180-4801-9244-f7f0bdff0cdd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:01.709366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25315b7-7180-4801-9244-f7f0bdff0cdd admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:01.759759 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1056/2105] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:01.770254 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1050/2106] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:21:01 2022] POST /identity/v3/users => generated 293 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:21:01.770493 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da21a555-6f86-4290-969a-e0975f2ffaf8 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:01.770690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da21a555-6f86-4290-969a-e0975f2ffaf8 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:01.771131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da21a555-6f86-4290-969a-e0975f2ffaf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:01.773914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25315b7-7180-4801-9244-f7f0bdff0cdd admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:01.776605 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c25315b7-7180-4801-9244-f7f0bdff0cdd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:01.792465 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25315b7-7180-4801-9244-f7f0bdff0cdd admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:01.804970 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c78ca2-4b7a-4d87-ac86-b2cd6df9638b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:01.805581 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c78ca2-4b7a-4d87-ac86-b2cd6df9638b admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:01.805973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c78ca2-4b7a-4d87-ac86-b2cd6df9638b admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:01.806620 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c78ca2-4b7a-4d87-ac86-b2cd6df9638b admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:01.807111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c78ca2-4b7a-4d87-ac86-b2cd6df9638b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:01.807402 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-da21a555-6f86-4290-969a-e0975f2ffaf8 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:01.809546 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c78ca2-4b7a-4d87-ac86-b2cd6df9638b admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:01.821688 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-da21a555-6f86-4290-969a-e0975f2ffaf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:01.827382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1051/2107] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:01 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:21:01.828372 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c78ca2-4b7a-4d87-ac86-b2cd6df9638b admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:01.838839 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c78ca2-4b7a-4d87-ac86-b2cd6df9638b admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:01.847956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458ea8c1-ead0-41a2-a7ad-71e3fa47eede admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:01.848656 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458ea8c1-ead0-41a2-a7ad-71e3fa47eede admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:01.849836 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458ea8c1-ead0-41a2-a7ad-71e3fa47eede admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:01.850193 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458ea8c1-ead0-41a2-a7ad-71e3fa47eede admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:01.850890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458ea8c1-ead0-41a2-a7ad-71e3fa47eede admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:01.854233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458ea8c1-ead0-41a2-a7ad-71e3fa47eede admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:01.859216 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1057/2108] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:01.863747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da21a555-6f86-4290-969a-e0975f2ffaf8 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:01.886172 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1052/2109] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:01 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:21:01.888173 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da21a555-6f86-4290-969a-e0975f2ffaf8 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:01.889374 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458ea8c1-ead0-41a2-a7ad-71e3fa47eede admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:01.897776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a02d99b-5332-4ee6-9813-0b1cb4cc1524 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:01.898392 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a02d99b-5332-4ee6-9813-0b1cb4cc1524 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:01.898674 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a02d99b-5332-4ee6-9813-0b1cb4cc1524 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:01.898942 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a02d99b-5332-4ee6-9813-0b1cb4cc1524 admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:01.899608 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a02d99b-5332-4ee6-9813-0b1cb4cc1524 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:21:01.900034 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a02d99b-5332-4ee6-9813-0b1cb4cc1524 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:01.906366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a02d99b-5332-4ee6-9813-0b1cb4cc1524 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:01.914920 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458ea8c1-ead0-41a2-a7ad-71e3fa47eede admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:01.924357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b64491d-f849-45db-b830-27278faffc4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:01.925039 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b64491d-f849-45db-b830-27278faffc4b admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:01.925324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b64491d-f849-45db-b830-27278faffc4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:01.925598 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b64491d-f849-45db-b830-27278faffc4b admin admin] PATH_INFO: `/v3/projects/8d46e7923d634ee8afcb6f24a2fb02e0/users/51872bb721114394b4b53d7c81969d7f/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:01.934513 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1058/2110] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:21:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:21:01.939341 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a02d99b-5332-4ee6-9813-0b1cb4cc1524 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:01.956333 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b64491d-f849-45db-b830-27278faffc4b admin admin] RBAC: Authorizing `identity:create_grant(project_id=8d46e7923d634ee8afcb6f24a2fb02e0, user_id=51872bb721114394b4b53d7c81969d7f, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:01.957331 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a02d99b-5332-4ee6-9813-0b1cb4cc1524 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:01.963963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b64491d-f849-45db-b830-27278faffc4b admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:01.975468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3239367-eb21-40b5-8d96-bc53e072fd9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:01.976196 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3239367-eb21-40b5-8d96-bc53e072fd9d admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:01.976456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3239367-eb21-40b5-8d96-bc53e072fd9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:01.976737 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3239367-eb21-40b5-8d96-bc53e072fd9d admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:01.977377 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3239367-eb21-40b5-8d96-bc53e072fd9d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:01.979316 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3239367-eb21-40b5-8d96-bc53e072fd9d admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:01.983151 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1053/2111] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:21:01 2022] PUT /identity/v3/projects/8d46e7923d634ee8afcb6f24a2fb02e0/users/51872bb721114394b4b53d7c81969d7f/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:01.994941 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e076a92d-5319-40a5-8675-03b2087198b3 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:01.995251 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e076a92d-5319-40a5-8675-03b2087198b3 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:01.995503 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e076a92d-5319-40a5-8675-03b2087198b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:02.033136 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1059/2112] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:21:01 2022] POST /identity/v3/projects => generated 369 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:21:02.036968 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3239367-eb21-40b5-8d96-bc53e072fd9d admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:02.066197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-e076a92d-5319-40a5-8675-03b2087198b3 None None] MFA Rules not processed for user `51872bb721114394b4b53d7c81969d7f`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:02.067596 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-e076a92d-5319-40a5-8675-03b2087198b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:02.084565 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=90,keystone:DELETE=4,keystone:INSERT=6 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:02.197285 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3239367-eb21-40b5-8d96-bc53e072fd9d admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:02.214019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1054/2113] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:01 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:21:02.218733 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c23971-b1b6-48df-8ccd-b19c7e695233 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:02.219463 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c23971-b1b6-48df-8ccd-b19c7e695233 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:02.220142 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c23971-b1b6-48df-8ccd-b19c7e695233 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:02.220142 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c23971-b1b6-48df-8ccd-b19c7e695233 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:02.220665 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c23971-b1b6-48df-8ccd-b19c7e695233 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:02.223086 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c23971-b1b6-48df-8ccd-b19c7e695233 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:02.231644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e076a92d-5319-40a5-8675-03b2087198b3 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:02.234326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-e076a92d-5319-40a5-8675-03b2087198b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:02.258768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1060/2114] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:21:02 2022] POST /identity/v3/users => generated 285 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:21:02.259867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c23971-b1b6-48df-8ccd-b19c7e695233 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:02.272252 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c23971-b1b6-48df-8ccd-b19c7e695233 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:02.282803 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1423e7-68d4-4480-92b4-f64004b8417b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:02.283898 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1423e7-68d4-4480-92b4-f64004b8417b admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:02.284188 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1423e7-68d4-4480-92b4-f64004b8417b admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:02.284603 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1423e7-68d4-4480-92b4-f64004b8417b admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:02.285316 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1423e7-68d4-4480-92b4-f64004b8417b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:02.287908 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1423e7-68d4-4480-92b4-f64004b8417b admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:02.310214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1061/2115] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:02 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:21:02.316825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1423e7-68d4-4480-92b4-f64004b8417b admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:02.336255 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1423e7-68d4-4480-92b4-f64004b8417b admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:02.338092 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e076a92d-5319-40a5-8675-03b2087198b3 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:02.344334 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22855ade-4f8e-409f-9d95-f14ca25cc0d4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:02.345453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22855ade-4f8e-409f-9d95-f14ca25cc0d4 service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:02.352419 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22855ade-4f8e-409f-9d95-f14ca25cc0d4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:02.352999 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22855ade-4f8e-409f-9d95-f14ca25cc0d4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:02.357093 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee71bc5a-88c1-4641-aac0-fa447f212763 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:02.357798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee71bc5a-88c1-4641-aac0-fa447f212763 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:02.358091 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee71bc5a-88c1-4641-aac0-fa447f212763 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:02.358938 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee71bc5a-88c1-4641-aac0-fa447f212763 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:02.359590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee71bc5a-88c1-4641-aac0-fa447f212763 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:02.361604 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee71bc5a-88c1-4641-aac0-fa447f212763 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:02.369013 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22855ade-4f8e-409f-9d95-f14ca25cc0d4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:02.370835 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22855ade-4f8e-409f-9d95-f14ca25cc0d4 service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:02.380067 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1062/2116] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:02 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:21:02.381229 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee71bc5a-88c1-4641-aac0-fa447f212763 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:02.388461 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1055/2117] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:02 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:21:02.393390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee71bc5a-88c1-4641-aac0-fa447f212763 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:02.409104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86764a5-4bd6-4a4d-a0b5-22732d36a4e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:02.409731 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86764a5-4bd6-4a4d-a0b5-22732d36a4e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:02.410061 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86764a5-4bd6-4a4d-a0b5-22732d36a4e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:02.411829 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86764a5-4bd6-4a4d-a0b5-22732d36a4e7 admin admin] PATH_INFO: `/v3/projects/5ada2c71f3c948f58cec85e4edaf1a92/users/887a8cde38b144709b9ded8adcaeedbe/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:02.422175 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22855ade-4f8e-409f-9d95-f14ca25cc0d4 service cinder] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:02.441966 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c86764a5-4bd6-4a4d-a0b5-22732d36a4e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5ada2c71f3c948f58cec85e4edaf1a92, user_id=887a8cde38b144709b9ded8adcaeedbe, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:02.443845 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22855ade-4f8e-409f-9d95-f14ca25cc0d4 service cinder] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:02.445301 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c86764a5-4bd6-4a4d-a0b5-22732d36a4e7 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:02.449334 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:INSERT=5,keystone:DELETE=1,keystone:SELECT=94 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:02.449682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e0adab-62f4-4a05-bce4-da529c55d51d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:02.450421 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e0adab-62f4-4a05-bce4-da529c55d51d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:02.450768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e0adab-62f4-4a05-bce4-da529c55d51d admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:02.451118 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e0adab-62f4-4a05-bce4-da529c55d51d admin admin] PATH_INFO: `/v3/users/51872bb721114394b4b53d7c81969d7f` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:02.452926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e0adab-62f4-4a05-bce4-da529c55d51d admin admin] RBAC: Authorizing `identity:delete_user(user_id=51872bb721114394b4b53d7c81969d7f)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:02.454981 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e0adab-62f4-4a05-bce4-da529c55d51d admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:02.471825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1063/2118] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:21:02 2022] PUT /identity/v3/projects/5ada2c71f3c948f58cec85e4edaf1a92/users/887a8cde38b144709b9ded8adcaeedbe/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:02.485595 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42de7ab4-a3af-4e3e-bf44-767d9caa318c None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:02.485823 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42de7ab4-a3af-4e3e-bf44-767d9caa318c None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:02.486313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42de7ab4-a3af-4e3e-bf44-767d9caa318c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:02.519078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-42de7ab4-a3af-4e3e-bf44-767d9caa318c None None] MFA Rules not processed for user `887a8cde38b144709b9ded8adcaeedbe`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:02.520401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-42de7ab4-a3af-4e3e-bf44-767d9caa318c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:02.528353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-47e0adab-62f4-4a05-bce4-da529c55d51d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '51872bb721114394b4b53d7c81969d7f'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:02.545995 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-47e0adab-62f4-4a05-bce4-da529c55d51d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '51872bb721114394b4b53d7c81969d7f'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:02.547286 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-47e0adab-62f4-4a05-bce4-da529c55d51d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '51872bb721114394b4b53d7c81969d7f'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:02.563107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-47e0adab-62f4-4a05-bce4-da529c55d51d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '51872bb721114394b4b53d7c81969d7f'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:02.590743 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-47e0adab-62f4-4a05-bce4-da529c55d51d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '51872bb721114394b4b53d7c81969d7f'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:02.600572 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1056/2119] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:21:02 2022] DELETE /identity/v3/users/51872bb721114394b4b53d7c81969d7f => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:21:02.613903 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e5716d-507b-465e-9b0f-75237118bada None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:02.614293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e5716d-507b-465e-9b0f-75237118bada None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:02.615428 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e5716d-507b-465e-9b0f-75237118bada None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:02.621350 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1064/2120] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:02 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:02.637787 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42de7ab4-a3af-4e3e-bf44-767d9caa318c None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:02.637787 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-42de7ab4-a3af-4e3e-bf44-767d9caa318c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:02.646231 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-71e5716d-507b-465e-9b0f-75237118bada None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:02.647481 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-71e5716d-507b-465e-9b0f-75237118bada None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:02.683382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:02.730167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42de7ab4-a3af-4e3e-bf44-767d9caa318c None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:02.763902 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1057/2121] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:21:02.806475 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce452abf-205c-4268-b62e-ac8dc5640627 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dcc53332e6b462baf349a72b19d2b3b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:02.807096 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce452abf-205c-4268-b62e-ac8dc5640627 service glance] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:02.807374 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce452abf-205c-4268-b62e-ac8dc5640627 service glance] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:02.807649 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce452abf-205c-4268-b62e-ac8dc5640627 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:02.838318 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e5716d-507b-465e-9b0f-75237118bada None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:02.839903 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-71e5716d-507b-465e-9b0f-75237118bada None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:02.941126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:02.953767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce452abf-205c-4268-b62e-ac8dc5640627 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:02.957565 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce452abf-205c-4268-b62e-ac8dc5640627 service glance] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:02.978924 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e5716d-507b-465e-9b0f-75237118bada None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:02.979559 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1065/2122] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:02 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:21:02.985351 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce452abf-205c-4268-b62e-ac8dc5640627 service glance] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:02.987470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ce452abf-205c-4268-b62e-ac8dc5640627 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:02.991124 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35a8723-e647-4f59-8d3c-499416f5dc3e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:02.991742 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35a8723-e647-4f59-8d3c-499416f5dc3e service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:02.992017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35a8723-e647-4f59-8d3c-499416f5dc3e service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:02.992282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35a8723-e647-4f59-8d3c-499416f5dc3e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:03.016499 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35a8723-e647-4f59-8d3c-499416f5dc3e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:03.018000 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35a8723-e647-4f59-8d3c-499416f5dc3e service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:03.047075 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1058/2123] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 209 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:21:03.101103 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce452abf-205c-4268-b62e-ac8dc5640627 service glance] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:03.114354 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7c3959-7592-4a05-89b6-e0d334a9df17 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:03.114997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7c3959-7592-4a05-89b6-e0d334a9df17 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:03.116976 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7c3959-7592-4a05-89b6-e0d334a9df17 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:03.117264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7c3959-7592-4a05-89b6-e0d334a9df17 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:03.122955 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ad7c3959-7592-4a05-89b6-e0d334a9df17 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:03.276138 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7c3959-7592-4a05-89b6-e0d334a9df17 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:03.277935 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7c3959-7592-4a05-89b6-e0d334a9df17 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:03.305827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1066/2124] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:02 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:21:04.001006 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5417b043-4637-4559-8ae7-5d8074930156 None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:04.001570 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5417b043-4637-4559-8ae7-5d8074930156 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:04.001853 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5417b043-4637-4559-8ae7-5d8074930156 None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:04.003426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1059/2125] 10.222.0.26 () {54 vars in 921 bytes} [Tue Sep 27 04:21:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:21:04.015326 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12880b32-cdde-4392-bc66-28b27d11ce81 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:04.015887 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12880b32-cdde-4392-bc66-28b27d11ce81 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:04.015887 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12880b32-cdde-4392-bc66-28b27d11ce81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:04.096367 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-12880b32-cdde-4392-bc66-28b27d11ce81 None None] MFA Rules not processed for user `3a1bc198dfc0486e9e56d4fb28cda05d`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:04.098122 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-12880b32-cdde-4392-bc66-28b27d11ce81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:04.242320 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5417b043-4637-4559-8ae7-5d8074930156 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:04.245092 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-5417b043-4637-4559-8ae7-5d8074930156 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:04.277252 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5417b043-4637-4559-8ae7-5d8074930156 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:04.282591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1067/2126] 10.222.0.26 () {58 vars in 1018 bytes} [Tue Sep 27 04:21:04 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 270 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:04.296929 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0272f2-904f-426c-a20c-2c2ab8f5a0f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:04.297865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0272f2-904f-426c-a20c-2c2ab8f5a0f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:04.298243 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0272f2-904f-426c-a20c-2c2ab8f5a0f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:04.298609 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0272f2-904f-426c-a20c-2c2ab8f5a0f6 admin admin] PATH_INFO: `/v3/projects/8d46e7923d634ee8afcb6f24a2fb02e0` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:04.303431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0272f2-904f-426c-a20c-2c2ab8f5a0f6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8d46e7923d634ee8afcb6f24a2fb02e0)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:04.306263 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0272f2-904f-426c-a20c-2c2ab8f5a0f6 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:04.327056 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12880b32-cdde-4392-bc66-28b27d11ce81 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:04.328644 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-12880b32-cdde-4392-bc66-28b27d11ce81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:04.350145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12880b32-cdde-4392-bc66-28b27d11ce81 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:04.357978 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b127800c-766b-4eb2-a29f-6d961fe24a26 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3a1bc198dfc0486e9e56d4fb28cda05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:04.358862 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b127800c-766b-4eb2-a29f-6d961fe24a26 service placement] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:04.359311 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b127800c-766b-4eb2-a29f-6d961fe24a26 service placement] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:04.359826 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b127800c-766b-4eb2-a29f-6d961fe24a26 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:04.376501 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b127800c-766b-4eb2-a29f-6d961fe24a26 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:04.378836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b127800c-766b-4eb2-a29f-6d961fe24a26 service placement] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:04.392020 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1068/2127] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:04 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:21:04.423064 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-6b0272f2-904f-426c-a20c-2c2ab8f5a0f6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8d46e7923d634ee8afcb6f24a2fb02e0', 'request_id': 'req-6b0272f2-904f-426c-a20c-2c2ab8f5a0f6'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:04.433588 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-6b0272f2-904f-426c-a20c-2c2ab8f5a0f6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8d46e7923d634ee8afcb6f24a2fb02e0', 'request_id': 'req-6b0272f2-904f-426c-a20c-2c2ab8f5a0f6'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:04.447179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-6b0272f2-904f-426c-a20c-2c2ab8f5a0f6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8d46e7923d634ee8afcb6f24a2fb02e0', 'request_id': 'req-6b0272f2-904f-426c-a20c-2c2ab8f5a0f6'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:04.471937 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-6b0272f2-904f-426c-a20c-2c2ab8f5a0f6 admin admin] The token cache is being invalidate because project 8d46e7923d634ee8afcb6f24a2fb02e0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:21:04.472337 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-6b0272f2-904f-426c-a20c-2c2ab8f5a0f6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:04.473086 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-6b0272f2-904f-426c-a20c-2c2ab8f5a0f6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:04.477783 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1060/2128] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:21:04 2022] DELETE /identity/v3/projects/8d46e7923d634ee8afcb6f24a2fb02e0 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:21:04.480605 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb0e727-5837-4ff6-a1cd-c19c65105770 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:04.480888 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb0e727-5837-4ff6-a1cd-c19c65105770 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:04.481182 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb0e727-5837-4ff6-a1cd-c19c65105770 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:04.503268 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-ebb0e727-5837-4ff6-a1cd-c19c65105770 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:04.504649 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ebb0e727-5837-4ff6-a1cd-c19c65105770 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:04.550272 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:04.634901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1069/2129] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:21:04.643599 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76344b7f-21d5-4d97-b83e-f9c6d98c1b8a None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:04.644481 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76344b7f-21d5-4d97-b83e-f9c6d98c1b8a None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:04.644481 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76344b7f-21d5-4d97-b83e-f9c6d98c1b8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:04.679133 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-76344b7f-21d5-4d97-b83e-f9c6d98c1b8a None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:04.680349 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-76344b7f-21d5-4d97-b83e-f9c6d98c1b8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:04.693207 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1061/2130] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:04.694258 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb0e727-5837-4ff6-a1cd-c19c65105770 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:04.704469 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb0e727-5837-4ff6-a1cd-c19c65105770 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:04.709395 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0726798-37e9-4891-8e30-328113d3694f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:04.710299 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0726798-37e9-4891-8e30-328113d3694f admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:04.710572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0726798-37e9-4891-8e30-328113d3694f admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:04.710890 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0726798-37e9-4891-8e30-328113d3694f admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:04.711541 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0726798-37e9-4891-8e30-328113d3694f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:21:04.711950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0726798-37e9-4891-8e30-328113d3694f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:04.714202 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0726798-37e9-4891-8e30-328113d3694f admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:04.734495 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1070/2131] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:21:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:21:04.738898 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76344b7f-21d5-4d97-b83e-f9c6d98c1b8a None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:04.755270 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76344b7f-21d5-4d97-b83e-f9c6d98c1b8a None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:04.773221 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b92e06-3f7f-438b-8ba7-1a42b2ab0f90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:04.773485 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b92e06-3f7f-438b-8ba7-1a42b2ab0f90 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:04.773872 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b92e06-3f7f-438b-8ba7-1a42b2ab0f90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:04.774071 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b92e06-3f7f-438b-8ba7-1a42b2ab0f90 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:04.774887 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b92e06-3f7f-438b-8ba7-1a42b2ab0f90 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:04.776853 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b92e06-3f7f-438b-8ba7-1a42b2ab0f90 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:04.814125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1062/2132] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:21:04 2022] POST /identity/v3/projects => generated 375 bytes in 76 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:21:04.819247 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0726798-37e9-4891-8e30-328113d3694f admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:04.837228 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0726798-37e9-4891-8e30-328113d3694f admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:05.033046 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b92e06-3f7f-438b-8ba7-1a42b2ab0f90 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:05.035212 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-90b92e06-3f7f-438b-8ba7-1a42b2ab0f90 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:05.056884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a3b92f-2478-4628-9d2e-abae6e20b78a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:05.057534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a3b92f-2478-4628-9d2e-abae6e20b78a admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:05.074278 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a3b92f-2478-4628-9d2e-abae6e20b78a admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:05.074693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a3b92f-2478-4628-9d2e-abae6e20b78a admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:05.075569 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a3b92f-2478-4628-9d2e-abae6e20b78a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:05.077379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a3b92f-2478-4628-9d2e-abae6e20b78a admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:05.120786 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:05.133887 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1071/2133] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:21:04 2022] POST /identity/v3/users => generated 288 bytes in 316 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:21:05.136393 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a3b92f-2478-4628-9d2e-abae6e20b78a admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:05.162161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a3b92f-2478-4628-9d2e-abae6e20b78a admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:05.169043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d06bb4-3205-4419-825a-202494014688 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:05.169891 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d06bb4-3205-4419-825a-202494014688 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:05.170299 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d06bb4-3205-4419-825a-202494014688 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:05.171554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d06bb4-3205-4419-825a-202494014688 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:05.173454 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d06bb4-3205-4419-825a-202494014688 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:05.178919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d06bb4-3205-4419-825a-202494014688 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:05.212537 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b92e06-3f7f-438b-8ba7-1a42b2ab0f90 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:05.219229 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1072/2134] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:05 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:21:05.219388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc3a625-a192-4df1-8dfb-466e746898b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:05.219388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc3a625-a192-4df1-8dfb-466e746898b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:05.219388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc3a625-a192-4df1-8dfb-466e746898b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:05.219658 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc3a625-a192-4df1-8dfb-466e746898b9 admin admin] PATH_INFO: `/v3/users/d4abb7018fa64528ad601dadb8123aa1` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:05.222494 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc3a625-a192-4df1-8dfb-466e746898b9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d4abb7018fa64528ad601dadb8123aa1)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:05.223055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d06bb4-3205-4419-825a-202494014688 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:05.224417 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc3a625-a192-4df1-8dfb-466e746898b9 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:05.249384 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d06bb4-3205-4419-825a-202494014688 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:05.270540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e002f3-9778-419f-9907-bd1bfc71460f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:05.274016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e002f3-9778-419f-9907-bd1bfc71460f admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:05.274376 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e002f3-9778-419f-9907-bd1bfc71460f admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:05.274726 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e002f3-9778-419f-9907-bd1bfc71460f admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:05.275469 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e002f3-9778-419f-9907-bd1bfc71460f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:05.279978 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e002f3-9778-419f-9907-bd1bfc71460f admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:05.309793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1073/2135] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:05 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:21:05.313177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e002f3-9778-419f-9907-bd1bfc71460f admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:05.337081 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e002f3-9778-419f-9907-bd1bfc71460f admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:05.348482 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca756584-e847-45e2-853e-728d3a1ac61b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:05.348482 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca756584-e847-45e2-853e-728d3a1ac61b admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:05.348482 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca756584-e847-45e2-853e-728d3a1ac61b admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:05.348482 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca756584-e847-45e2-853e-728d3a1ac61b admin admin] PATH_INFO: `/v3/projects/8ebcf45fe4054d38839569d76df9565f/users/2e505b67d0064ff2a255c2d4cc47248c/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:05.362278 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-6cc3a625-a192-4df1-8dfb-466e746898b9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd4abb7018fa64528ad601dadb8123aa1'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:05.377672 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-6cc3a625-a192-4df1-8dfb-466e746898b9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd4abb7018fa64528ad601dadb8123aa1'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:05.378594 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-6cc3a625-a192-4df1-8dfb-466e746898b9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd4abb7018fa64528ad601dadb8123aa1'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:05.380074 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca756584-e847-45e2-853e-728d3a1ac61b admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ebcf45fe4054d38839569d76df9565f, user_id=2e505b67d0064ff2a255c2d4cc47248c, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:05.383468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca756584-e847-45e2-853e-728d3a1ac61b admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:05.387593 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-6cc3a625-a192-4df1-8dfb-466e746898b9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd4abb7018fa64528ad601dadb8123aa1'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:05.402829 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1074/2136] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:21:05 2022] PUT /identity/v3/projects/8ebcf45fe4054d38839569d76df9565f/users/2e505b67d0064ff2a255c2d4cc47248c/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:05.416786 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7304bda0-34a7-417a-9eca-4f4d81b599d9 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:05.417067 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7304bda0-34a7-417a-9eca-4f4d81b599d9 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:05.418814 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-6cc3a625-a192-4df1-8dfb-466e746898b9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd4abb7018fa64528ad601dadb8123aa1'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:05.419366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7304bda0-34a7-417a-9eca-4f4d81b599d9 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:05.422260 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1075/2137] 10.222.0.26 () {54 vars in 921 bytes} [Tue Sep 27 04:21:05 2022] GET /identity => generated 270 bytes in 19 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:21:05.424172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ef903a-d122-4118-a9cf-78ce9f986d4b None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:05.424492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ef903a-d122-4118-a9cf-78ce9f986d4b None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:05.424738 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ef903a-d122-4118-a9cf-78ce9f986d4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:05.429147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1063/2138] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:21:05 2022] DELETE /identity/v3/users/d4abb7018fa64528ad601dadb8123aa1 => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:05.443926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d5f048-f460-4d59-9f9b-5c2999f22753 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:05.444078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d5f048-f460-4d59-9f9b-5c2999f22753 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:05.444603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d5f048-f460-4d59-9f9b-5c2999f22753 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:05.504511 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-d8d5f048-f460-4d59-9f9b-5c2999f22753 None None] MFA Rules not processed for user `3a1bc198dfc0486e9e56d4fb28cda05d`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:05.505794 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-d8d5f048-f460-4d59-9f9b-5c2999f22753 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:05.524194 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-12ef903a-d122-4118-a9cf-78ce9f986d4b None None] MFA Rules not processed for user `2e505b67d0064ff2a255c2d4cc47248c`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:05.529818 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-12ef903a-d122-4118-a9cf-78ce9f986d4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:05.601062 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:05.623115 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1064/2139] 10.222.0.26 () {58 vars in 1018 bytes} [Tue Sep 27 04:21:05 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:05.627183 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d5f048-f460-4d59-9f9b-5c2999f22753 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:05.629034 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-d8d5f048-f460-4d59-9f9b-5c2999f22753 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:05.635430 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1076/2140] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:05 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:05.636707 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ef903a-d122-4118-a9cf-78ce9f986d4b None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:05.638135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-12ef903a-d122-4118-a9cf-78ce9f986d4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:05.665485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ef903a-d122-4118-a9cf-78ce9f986d4b None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:05.671757 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b9ba59-409d-4cd7-8747-08672bb860ff service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3a1bc198dfc0486e9e56d4fb28cda05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:05.672616 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b9ba59-409d-4cd7-8747-08672bb860ff service placement] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:05.672914 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b9ba59-409d-4cd7-8747-08672bb860ff service placement] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:05.673191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b9ba59-409d-4cd7-8747-08672bb860ff service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:05.696390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b9ba59-409d-4cd7-8747-08672bb860ff service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:05.699960 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b9ba59-409d-4cd7-8747-08672bb860ff service placement] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:05.715512 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:05.722967 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1077/2141] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:05 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:21:05.723835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b9ba59-409d-4cd7-8747-08672bb860ff service placement] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:05.725144 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-13b9ba59-409d-4cd7-8747-08672bb860ff service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:05.750882 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d5f048-f460-4d59-9f9b-5c2999f22753 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:05.757606 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6437ff53-89aa-4a15-8962-0c3005bce1c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:05.758670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6437ff53-89aa-4a15-8962-0c3005bce1c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:05.758670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6437ff53-89aa-4a15-8962-0c3005bce1c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:05.759057 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6437ff53-89aa-4a15-8962-0c3005bce1c5 admin admin] PATH_INFO: `/v3/users/03912e6e8a074eb1814c90158c246281` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:05.760361 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6437ff53-89aa-4a15-8962-0c3005bce1c5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=03912e6e8a074eb1814c90158c246281)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:05.762432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6437ff53-89aa-4a15-8962-0c3005bce1c5 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:05.791671 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b9ba59-409d-4cd7-8747-08672bb860ff service placement] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:05.796670 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ea3515-924a-4cd0-9c49-9532c574c62f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:05.797364 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ea3515-924a-4cd0-9c49-9532c574c62f service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:05.797714 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ea3515-924a-4cd0-9c49-9532c574c62f service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:05.798089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ea3515-924a-4cd0-9c49-9532c574c62f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:05.807638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ea3515-924a-4cd0-9c49-9532c574c62f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:05.809056 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ea3515-924a-4cd0-9c49-9532c574c62f service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:05.815960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-6437ff53-89aa-4a15-8962-0c3005bce1c5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '03912e6e8a074eb1814c90158c246281'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:05.818468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1078/2142] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:05 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:21:05.819262 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ea3515-924a-4cd0-9c49-9532c574c62f service cinder] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:05.820602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-93ea3515-924a-4cd0-9c49-9532c574c62f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:05.824357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-6437ff53-89aa-4a15-8962-0c3005bce1c5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '03912e6e8a074eb1814c90158c246281'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:05.825228 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-6437ff53-89aa-4a15-8962-0c3005bce1c5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '03912e6e8a074eb1814c90158c246281'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:05.835907 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-6437ff53-89aa-4a15-8962-0c3005bce1c5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '03912e6e8a074eb1814c90158c246281'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:05.839224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ea3515-924a-4cd0-9c49-9532c574c62f service cinder] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:05.846009 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bbd2a72-cc1f-41f4-8c8e-c5433a81654b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:05.846956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbd2a72-cc1f-41f4-8c8e-c5433a81654b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:05.847239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbd2a72-cc1f-41f4-8c8e-c5433a81654b admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:05.847517 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbd2a72-cc1f-41f4-8c8e-c5433a81654b admin admin] PATH_INFO: `/v3/projects/1ffa7078248541158035d49ab4254b73` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:05.848866 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bbd2a72-cc1f-41f4-8c8e-c5433a81654b admin admin] RBAC: Authorizing `identity:delete_project(project_id=1ffa7078248541158035d49ab4254b73)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:05.850565 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-6437ff53-89aa-4a15-8962-0c3005bce1c5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '03912e6e8a074eb1814c90158c246281'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:05.850865 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bbd2a72-cc1f-41f4-8c8e-c5433a81654b admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:05.857588 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1065/2143] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:21:05 2022] DELETE /identity/v3/users/03912e6e8a074eb1814c90158c246281 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:05.910568 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-4bbd2a72-cc1f-41f4-8c8e-c5433a81654b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1ffa7078248541158035d49ab4254b73', 'request_id': 'req-4bbd2a72-cc1f-41f4-8c8e-c5433a81654b'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:05.918123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-4bbd2a72-cc1f-41f4-8c8e-c5433a81654b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1ffa7078248541158035d49ab4254b73', 'request_id': 'req-4bbd2a72-cc1f-41f4-8c8e-c5433a81654b'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:05.925664 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-4bbd2a72-cc1f-41f4-8c8e-c5433a81654b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1ffa7078248541158035d49ab4254b73', 'request_id': 'req-4bbd2a72-cc1f-41f4-8c8e-c5433a81654b'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:05.951818 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-4bbd2a72-cc1f-41f4-8c8e-c5433a81654b admin admin] The token cache is being invalidate because project 1ffa7078248541158035d49ab4254b73 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:21:05.952943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-4bbd2a72-cc1f-41f4-8c8e-c5433a81654b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:05.953865 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-4bbd2a72-cc1f-41f4-8c8e-c5433a81654b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:05.955670 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1079/2144] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:21:05 2022] DELETE /identity/v3/projects/1ffa7078248541158035d49ab4254b73 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:05.959536 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6437ff53-89aa-4a15-8962-0c3005bce1c5 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:05.961075 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-6437ff53-89aa-4a15-8962-0c3005bce1c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:06.056852 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6437ff53-89aa-4a15-8962-0c3005bce1c5 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:06.057568 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bbd2a72-cc1f-41f4-8c8e-c5433a81654b admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:06.058671 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-4bbd2a72-cc1f-41f4-8c8e-c5433a81654b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:06.137547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa1be9c-ec25-4550-a029-ae546dbaa332 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:06.138408 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa1be9c-ec25-4550-a029-ae546dbaa332 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:06.138725 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa1be9c-ec25-4550-a029-ae546dbaa332 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:06.138998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa1be9c-ec25-4550-a029-ae546dbaa332 admin admin] PATH_INFO: `/v3/users/04f0174445d0410a86de96a190d654cb` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:06.141188 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa1be9c-ec25-4550-a029-ae546dbaa332 admin admin] RBAC: Authorizing `identity:delete_user(user_id=04f0174445d0410a86de96a190d654cb)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:06.143982 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa1be9c-ec25-4550-a029-ae546dbaa332 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:06.163738 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bbd2a72-cc1f-41f4-8c8e-c5433a81654b admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:06.175386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b6cb1e-a472-4538-93ad-1b2476d6668e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:06.175541 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:06.176218 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b6cb1e-a472-4538-93ad-1b2476d6668e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:06.176374 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b6cb1e-a472-4538-93ad-1b2476d6668e admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:06.176638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b6cb1e-a472-4538-93ad-1b2476d6668e admin admin] PATH_INFO: `/v3/projects/7914c4b5217f411f807854ccb07e0365` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:06.182204 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b6cb1e-a472-4538-93ad-1b2476d6668e admin admin] RBAC: Authorizing `identity:delete_project(project_id=7914c4b5217f411f807854ccb07e0365)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:06.184115 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b6cb1e-a472-4538-93ad-1b2476d6668e admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:06.186057 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:06.234254 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-9fa1be9c-ec25-4550-a029-ae546dbaa332 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '04f0174445d0410a86de96a190d654cb'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:06.243665 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-9fa1be9c-ec25-4550-a029-ae546dbaa332 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '04f0174445d0410a86de96a190d654cb'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:06.244508 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-9fa1be9c-ec25-4550-a029-ae546dbaa332 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '04f0174445d0410a86de96a190d654cb'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:06.255430 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-f8b6cb1e-a472-4538-93ad-1b2476d6668e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7914c4b5217f411f807854ccb07e0365', 'request_id': 'req-f8b6cb1e-a472-4538-93ad-1b2476d6668e'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:06.260873 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-9fa1be9c-ec25-4550-a029-ae546dbaa332 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '04f0174445d0410a86de96a190d654cb'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:06.262241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-f8b6cb1e-a472-4538-93ad-1b2476d6668e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7914c4b5217f411f807854ccb07e0365', 'request_id': 'req-f8b6cb1e-a472-4538-93ad-1b2476d6668e'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:06.273853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-f8b6cb1e-a472-4538-93ad-1b2476d6668e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7914c4b5217f411f807854ccb07e0365', 'request_id': 'req-f8b6cb1e-a472-4538-93ad-1b2476d6668e'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:06.289246 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-9fa1be9c-ec25-4550-a029-ae546dbaa332 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '04f0174445d0410a86de96a190d654cb'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:06.298151 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1066/2145] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:21:05 2022] DELETE /identity/v3/users/04f0174445d0410a86de96a190d654cb => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:06.302669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-f8b6cb1e-a472-4538-93ad-1b2476d6668e admin admin] The token cache is being invalidate because project 7914c4b5217f411f807854ccb07e0365 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:21:06.303114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-f8b6cb1e-a472-4538-93ad-1b2476d6668e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:06.306168 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-f8b6cb1e-a472-4538-93ad-1b2476d6668e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:06.308843 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1080/2146] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:21:06 2022] DELETE /identity/v3/projects/7914c4b5217f411f807854ccb07e0365 => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:21:06.314191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec266d8-9465-4b4c-bbd3-1b6b2a677468 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:06.314480 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec266d8-9465-4b4c-bbd3-1b6b2a677468 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:06.314756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec266d8-9465-4b4c-bbd3-1b6b2a677468 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:06.334724 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-1ec266d8-9465-4b4c-bbd3-1b6b2a677468 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:06.336026 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-1ec266d8-9465-4b4c-bbd3-1b6b2a677468 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:06.399279 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1067/2147] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:06.408340 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ead33d3-3e88-4151-995d-8b86c0a9a415 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:06.408340 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ead33d3-3e88-4151-995d-8b86c0a9a415 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:06.408631 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ead33d3-3e88-4151-995d-8b86c0a9a415 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:06.429689 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-6ead33d3-3e88-4151-995d-8b86c0a9a415 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:06.430949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6ead33d3-3e88-4151-995d-8b86c0a9a415 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:06.439515 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1081/2148] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:06.443731 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec266d8-9465-4b4c-bbd3-1b6b2a677468 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:06.454547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec266d8-9465-4b4c-bbd3-1b6b2a677468 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:06.462075 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e515f55-80dd-40ae-aeeb-bed6dc9b8575 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:06.462693 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e515f55-80dd-40ae-aeeb-bed6dc9b8575 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:06.463125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e515f55-80dd-40ae-aeeb-bed6dc9b8575 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:06.463489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e515f55-80dd-40ae-aeeb-bed6dc9b8575 admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:06.464330 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e515f55-80dd-40ae-aeeb-bed6dc9b8575 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:21:06.464642 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e515f55-80dd-40ae-aeeb-bed6dc9b8575 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:06.466590 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e515f55-80dd-40ae-aeeb-bed6dc9b8575 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:06.484804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ead33d3-3e88-4151-995d-8b86c0a9a415 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:06.486486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1068/2149] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:21:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:21:06.490069 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6ead33d3-3e88-4151-995d-8b86c0a9a415 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:06.490318 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e515f55-80dd-40ae-aeeb-bed6dc9b8575 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:06.503130 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e515f55-80dd-40ae-aeeb-bed6dc9b8575 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:06.504614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ead33d3-3e88-4151-995d-8b86c0a9a415 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:06.508838 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9afb45-dbb2-4bc9-aff7-ed1d3c4a79c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:06.509455 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9afb45-dbb2-4bc9-aff7-ed1d3c4a79c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:06.509566 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e79778-3a4d-4237-ab1c-090022d778ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:06.509788 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9afb45-dbb2-4bc9-aff7-ed1d3c4a79c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:06.510039 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9afb45-dbb2-4bc9-aff7-ed1d3c4a79c3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:06.510307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e79778-3a4d-4237-ab1c-090022d778ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:06.510670 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e79778-3a4d-4237-ab1c-090022d778ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:06.510757 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc9afb45-dbb2-4bc9-aff7-ed1d3c4a79c3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:06.510857 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e79778-3a4d-4237-ab1c-090022d778ad admin admin] PATH_INFO: `/v3/projects/a5d68445c296413a86184878a98b563b` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:06.511922 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e79778-3a4d-4237-ab1c-090022d778ad admin admin] RBAC: Authorizing `identity:delete_project(project_id=a5d68445c296413a86184878a98b563b)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:06.512575 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc9afb45-dbb2-4bc9-aff7-ed1d3c4a79c3 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:06.513889 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e79778-3a4d-4237-ab1c-090022d778ad admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:06.547467 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1069/2150] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:21:06 2022] POST /identity/v3/projects => generated 393 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:21:06.548520 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9afb45-dbb2-4bc9-aff7-ed1d3c4a79c3 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:06.589640 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-d0e79778-3a4d-4237-ab1c-090022d778ad admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a5d68445c296413a86184878a98b563b', 'request_id': 'req-d0e79778-3a4d-4237-ab1c-090022d778ad'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:06.598623 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-d0e79778-3a4d-4237-ab1c-090022d778ad admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a5d68445c296413a86184878a98b563b', 'request_id': 'req-d0e79778-3a4d-4237-ab1c-090022d778ad'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:06.607439 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-d0e79778-3a4d-4237-ab1c-090022d778ad admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a5d68445c296413a86184878a98b563b', 'request_id': 'req-d0e79778-3a4d-4237-ab1c-090022d778ad'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:06.622837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-d0e79778-3a4d-4237-ab1c-090022d778ad admin admin] The token cache is being invalidate because project a5d68445c296413a86184878a98b563b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:21:06.623056 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-d0e79778-3a4d-4237-ab1c-090022d778ad admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:06.623647 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-d0e79778-3a4d-4237-ab1c-090022d778ad admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:06.625346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1082/2151] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:21:06 2022] DELETE /identity/v3/projects/a5d68445c296413a86184878a98b563b => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:06.630409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=9,keystone:INSERT=2 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:06.631977 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b5c771-5f19-4f59-97d1-c1e31736b855 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:06.632248 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9afb45-dbb2-4bc9-aff7-ed1d3c4a79c3 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:06.632366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b5c771-5f19-4f59-97d1-c1e31736b855 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:06.632647 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b5c771-5f19-4f59-97d1-c1e31736b855 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:06.657803 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-91b5c771-5f19-4f59-97d1-c1e31736b855 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:06.659041 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-91b5c771-5f19-4f59-97d1-c1e31736b855 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:06.700887 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4544d0-0cc3-4a50-82c9-19a96ded3007 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:06.701496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4544d0-0cc3-4a50-82c9-19a96ded3007 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:06.701803 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4544d0-0cc3-4a50-82c9-19a96ded3007 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:06.702112 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4544d0-0cc3-4a50-82c9-19a96ded3007 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:06.702744 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4544d0-0cc3-4a50-82c9-19a96ded3007 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:06.704673 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4544d0-0cc3-4a50-82c9-19a96ded3007 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:06.725562 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1083/2152] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:21:06.730783 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1070/2153] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:21:06 2022] POST /identity/v3/users => generated 297 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:21:06.732604 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b5c771-5f19-4f59-97d1-c1e31736b855 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:06.733143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e71dcfd-ac1e-4c3b-a7f2-ec45f00ebb59 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:06.734095 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e71dcfd-ac1e-4c3b-a7f2-ec45f00ebb59 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:06.734458 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e71dcfd-ac1e-4c3b-a7f2-ec45f00ebb59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:06.734594 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-91b5c771-5f19-4f59-97d1-c1e31736b855 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:06.746302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b5c771-5f19-4f59-97d1-c1e31736b855 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:06.751237 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0421205c-224c-459e-9bba-617b43932ddb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:06.751839 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0421205c-224c-459e-9bba-617b43932ddb admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:06.752117 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0421205c-224c-459e-9bba-617b43932ddb admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:06.752387 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0421205c-224c-459e-9bba-617b43932ddb admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:06.753026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0421205c-224c-459e-9bba-617b43932ddb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:06.754656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0421205c-224c-459e-9bba-617b43932ddb admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:06.755298 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-0e71dcfd-ac1e-4c3b-a7f2-ec45f00ebb59 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:06.755974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:06.756605 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0e71dcfd-ac1e-4c3b-a7f2-ec45f00ebb59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:06.766152 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1071/2154] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:06.769996 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1084/2155] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:06 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:21:06.770456 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e71dcfd-ac1e-4c3b-a7f2-ec45f00ebb59 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:06.770814 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0421205c-224c-459e-9bba-617b43932ddb admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:06.782691 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e71dcfd-ac1e-4c3b-a7f2-ec45f00ebb59 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:06.785644 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0421205c-224c-459e-9bba-617b43932ddb admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:06.789085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273d3b75-87f3-4175-af45-6094fff5a86a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:06.790175 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273d3b75-87f3-4175-af45-6094fff5a86a admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:06.790457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273d3b75-87f3-4175-af45-6094fff5a86a admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:06.790757 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273d3b75-87f3-4175-af45-6094fff5a86a admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:06.791315 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced10176-6dc2-4169-b0bf-f2999fffd824 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:06.791451 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273d3b75-87f3-4175-af45-6094fff5a86a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:21:06.791774 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273d3b75-87f3-4175-af45-6094fff5a86a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:06.791992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced10176-6dc2-4169-b0bf-f2999fffd824 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:06.792382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced10176-6dc2-4169-b0bf-f2999fffd824 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:06.792667 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced10176-6dc2-4169-b0bf-f2999fffd824 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:06.793416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced10176-6dc2-4169-b0bf-f2999fffd824 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:06.793549 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273d3b75-87f3-4175-af45-6094fff5a86a admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:06.795187 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced10176-6dc2-4169-b0bf-f2999fffd824 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:06.806951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1085/2156] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:06 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:21:06.811335 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1072/2157] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:21:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:21:06.811513 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced10176-6dc2-4169-b0bf-f2999fffd824 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:06.812582 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273d3b75-87f3-4175-af45-6094fff5a86a admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:06.821416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced10176-6dc2-4169-b0bf-f2999fffd824 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:06.822496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273d3b75-87f3-4175-af45-6094fff5a86a admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:06.826112 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef4e0be-88cd-4c73-8baf-4b805fabc3bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:06.826750 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef4e0be-88cd-4c73-8baf-4b805fabc3bc admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:06.827081 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef4e0be-88cd-4c73-8baf-4b805fabc3bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:06.827206 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131bf6c5-eab3-45d0-b9b7-e5fb6e830a88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:06.827361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef4e0be-88cd-4c73-8baf-4b805fabc3bc admin admin] PATH_INFO: `/v3/projects/c25f11f92d4e4fd8965fe630ad77f3c3/users/2d0ba2ae3ffe4bb9b99f1347719df411/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:06.827724 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131bf6c5-eab3-45d0-b9b7-e5fb6e830a88 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:06.828081 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131bf6c5-eab3-45d0-b9b7-e5fb6e830a88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:06.828567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131bf6c5-eab3-45d0-b9b7-e5fb6e830a88 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:06.829167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131bf6c5-eab3-45d0-b9b7-e5fb6e830a88 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:06.831417 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131bf6c5-eab3-45d0-b9b7-e5fb6e830a88 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:06.843525 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef4e0be-88cd-4c73-8baf-4b805fabc3bc admin admin] RBAC: Authorizing `identity:create_grant(project_id=c25f11f92d4e4fd8965fe630ad77f3c3, user_id=2d0ba2ae3ffe4bb9b99f1347719df411, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:06.846938 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef4e0be-88cd-4c73-8baf-4b805fabc3bc admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:06.863962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1073/2158] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:21:06 2022] POST /identity/v3/projects => generated 375 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:21:06.864144 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1086/2159] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:21:06 2022] PUT /identity/v3/projects/c25f11f92d4e4fd8965fe630ad77f3c3/users/2d0ba2ae3ffe4bb9b99f1347719df411/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:21:06.865350 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131bf6c5-eab3-45d0-b9b7-e5fb6e830a88 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:06.867248 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21176622-202d-4bfb-80ca-ffb27f997586 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:06.867584 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21176622-202d-4bfb-80ca-ffb27f997586 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:06.867883 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21176622-202d-4bfb-80ca-ffb27f997586 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:06.888218 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-21176622-202d-4bfb-80ca-ffb27f997586 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:06.889414 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-21176622-202d-4bfb-80ca-ffb27f997586 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:06.931572 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131bf6c5-eab3-45d0-b9b7-e5fb6e830a88 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:06.936194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d309ac0-c5bf-4e36-8c92-7cd333d1bb5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:06.936826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d309ac0-c5bf-4e36-8c92-7cd333d1bb5e admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:06.937113 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d309ac0-c5bf-4e36-8c92-7cd333d1bb5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:06.937389 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d309ac0-c5bf-4e36-8c92-7cd333d1bb5e admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:06.938056 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d309ac0-c5bf-4e36-8c92-7cd333d1bb5e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:06.939967 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d309ac0-c5bf-4e36-8c92-7cd333d1bb5e admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:06.956963 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1087/2160] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:06.971695 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1074/2161] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:21:06 2022] POST /identity/v3/users => generated 288 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:21:06.978213 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21176622-202d-4bfb-80ca-ffb27f997586 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:06.978428 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d309ac0-c5bf-4e36-8c92-7cd333d1bb5e admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:06.979761 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-21176622-202d-4bfb-80ca-ffb27f997586 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:06.989926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d309ac0-c5bf-4e36-8c92-7cd333d1bb5e admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:06.996291 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4e2837-aedb-40d8-922b-641f6f0bd18f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:06.996898 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4e2837-aedb-40d8-922b-641f6f0bd18f admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:06.997179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4e2837-aedb-40d8-922b-641f6f0bd18f admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:06.997462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4e2837-aedb-40d8-922b-641f6f0bd18f admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:06.998133 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4e2837-aedb-40d8-922b-641f6f0bd18f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:06.999819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4e2837-aedb-40d8-922b-641f6f0bd18f admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:07.010009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1075/2162] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:06 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:21:07.013876 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4e2837-aedb-40d8-922b-641f6f0bd18f admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:07.024480 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4e2837-aedb-40d8-922b-641f6f0bd18f admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:07.031146 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac76aee-1cac-488c-ad15-9b48e7976ea3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:07.031736 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac76aee-1cac-488c-ad15-9b48e7976ea3 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:07.032017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac76aee-1cac-488c-ad15-9b48e7976ea3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:07.032290 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac76aee-1cac-488c-ad15-9b48e7976ea3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:07.032923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac76aee-1cac-488c-ad15-9b48e7976ea3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:07.034537 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac76aee-1cac-488c-ad15-9b48e7976ea3 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:07.042042 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21176622-202d-4bfb-80ca-ffb27f997586 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:07.045632 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd05210b-f55c-44db-b4f0-48a6371454f3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:07.046312 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd05210b-f55c-44db-b4f0-48a6371454f3 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:07.046593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd05210b-f55c-44db-b4f0-48a6371454f3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:07.046712 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1076/2163] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:07 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:21:07.046938 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd05210b-f55c-44db-b4f0-48a6371454f3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:07.050609 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac76aee-1cac-488c-ad15-9b48e7976ea3 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:07.056173 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd05210b-f55c-44db-b4f0-48a6371454f3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:07.057587 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd05210b-f55c-44db-b4f0-48a6371454f3 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:07.059973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac76aee-1cac-488c-ad15-9b48e7976ea3 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:07.065136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da60430-038e-4bef-875a-9b5c588c8ca2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:07.065777 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da60430-038e-4bef-875a-9b5c588c8ca2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:07.066066 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da60430-038e-4bef-875a-9b5c588c8ca2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:07.066338 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da60430-038e-4bef-875a-9b5c588c8ca2 admin admin] PATH_INFO: `/v3/projects/118a051914be4932887e976dab9efdf5/users/f49663f1fd5a47b5bce5cee57ce2f4b0/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:07.067364 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1088/2164] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:21:07.087543 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da60430-038e-4bef-875a-9b5c588c8ca2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=118a051914be4932887e976dab9efdf5, user_id=f49663f1fd5a47b5bce5cee57ce2f4b0, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:07.090283 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da60430-038e-4bef-875a-9b5c588c8ca2 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:07.093228 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:07.106453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1077/2165] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:21:07 2022] PUT /identity/v3/projects/118a051914be4932887e976dab9efdf5/users/f49663f1fd5a47b5bce5cee57ce2f4b0/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:21:07.117633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18516f56-5f8b-49b7-8a3d-88b0a3f59f68 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:07.117992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18516f56-5f8b-49b7-8a3d-88b0a3f59f68 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:07.118274 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18516f56-5f8b-49b7-8a3d-88b0a3f59f68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:07.149171 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-18516f56-5f8b-49b7-8a3d-88b0a3f59f68 None None] MFA Rules not processed for user `f49663f1fd5a47b5bce5cee57ce2f4b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:07.150401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-18516f56-5f8b-49b7-8a3d-88b0a3f59f68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:07.220096 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:07.236060 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1089/2166] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:07 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:07.247385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da60430-038e-4bef-875a-9b5c588c8ca2 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:07.249045 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-6da60430-038e-4bef-875a-9b5c588c8ca2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:07.351450 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da60430-038e-4bef-875a-9b5c588c8ca2 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:07.355242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe38b995-af76-430a-8547-ce24067f1e44 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dcc53332e6b462baf349a72b19d2b3b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:07.355910 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe38b995-af76-430a-8547-ce24067f1e44 service glance] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:07.356116 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe38b995-af76-430a-8547-ce24067f1e44 service glance] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:07.356390 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe38b995-af76-430a-8547-ce24067f1e44 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:07.365398 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe38b995-af76-430a-8547-ce24067f1e44 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:07.366835 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe38b995-af76-430a-8547-ce24067f1e44 service glance] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:07.377652 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1078/2167] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:07 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:21:11.305041 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff707f35-4ed8-4dc1-a1ac-29108012f5ed None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:11.305567 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff707f35-4ed8-4dc1-a1ac-29108012f5ed None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:11.305567 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff707f35-4ed8-4dc1-a1ac-29108012f5ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:11.334615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-ff707f35-4ed8-4dc1-a1ac-29108012f5ed None None] MFA Rules not processed for user `2d0ba2ae3ffe4bb9b99f1347719df411`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:11.335797 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ff707f35-4ed8-4dc1-a1ac-29108012f5ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:11.401690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1090/2168] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:11 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:11.421979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7b938f-839e-4c40-b4de-0df485447bb0 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:11.421979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7b938f-839e-4c40-b4de-0df485447bb0 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:11.421979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7b938f-839e-4c40-b4de-0df485447bb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:11.451716 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-ff7b938f-839e-4c40-b4de-0df485447bb0 None None] MFA Rules not processed for user `2d0ba2ae3ffe4bb9b99f1347719df411`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:11.452942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ff7b938f-839e-4c40-b4de-0df485447bb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:11.459845 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1079/2169] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:11 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:11.472340 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff707f35-4ed8-4dc1-a1ac-29108012f5ed None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:11.476014 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ff707f35-4ed8-4dc1-a1ac-29108012f5ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:11.589973 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff707f35-4ed8-4dc1-a1ac-29108012f5ed None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:11.597600 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328fd76b-f2d4-4778-a3d6-91151d9ae2f3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:11.598259 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328fd76b-f2d4-4778-a3d6-91151d9ae2f3 service nova] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:11.598535 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328fd76b-f2d4-4778-a3d6-91151d9ae2f3 service nova] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:11.598805 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328fd76b-f2d4-4778-a3d6-91151d9ae2f3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:11.612127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328fd76b-f2d4-4778-a3d6-91151d9ae2f3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:11.614183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328fd76b-f2d4-4778-a3d6-91151d9ae2f3 service nova] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:11.626004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1091/2170] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:11 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:21:12.053855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7b938f-839e-4c40-b4de-0df485447bb0 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:12.055290 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ff7b938f-839e-4c40-b4de-0df485447bb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:12.148838 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7b938f-839e-4c40-b4de-0df485447bb0 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:12.154993 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b71e04-5e68-444b-9e99-dc9c0cfd1f43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:12.154993 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b71e04-5e68-444b-9e99-dc9c0cfd1f43 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:12.154993 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b71e04-5e68-444b-9e99-dc9c0cfd1f43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:12.155945 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b71e04-5e68-444b-9e99-dc9c0cfd1f43 admin admin] PATH_INFO: `/v3/users/24224702d2724959a69713c69630384c` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:12.156946 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b71e04-5e68-444b-9e99-dc9c0cfd1f43 admin admin] RBAC: Authorizing `identity:delete_user(user_id=24224702d2724959a69713c69630384c)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:12.158817 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b71e04-5e68-444b-9e99-dc9c0cfd1f43 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:12.245565 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-96b71e04-5e68-444b-9e99-dc9c0cfd1f43 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '24224702d2724959a69713c69630384c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:12.254380 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-96b71e04-5e68-444b-9e99-dc9c0cfd1f43 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '24224702d2724959a69713c69630384c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:12.255459 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-96b71e04-5e68-444b-9e99-dc9c0cfd1f43 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '24224702d2724959a69713c69630384c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:12.267983 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-96b71e04-5e68-444b-9e99-dc9c0cfd1f43 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '24224702d2724959a69713c69630384c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:12.286792 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-96b71e04-5e68-444b-9e99-dc9c0cfd1f43 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '24224702d2724959a69713c69630384c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:12.294494 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1080/2171] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:21:12 2022] DELETE /identity/v3/users/24224702d2724959a69713c69630384c => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:12.298878 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c2822e-8cc7-423d-a9a0-b19af3777861 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:12.299163 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c2822e-8cc7-423d-a9a0-b19af3777861 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:12.299435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c2822e-8cc7-423d-a9a0-b19af3777861 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:12.330228 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-45c2822e-8cc7-423d-a9a0-b19af3777861 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:12.331611 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-45c2822e-8cc7-423d-a9a0-b19af3777861 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:12.438382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1092/2172] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:12.447999 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b71e04-5e68-444b-9e99-dc9c0cfd1f43 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:12.449429 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-96b71e04-5e68-444b-9e99-dc9c0cfd1f43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:12.463796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:INSERT=2,keystone:SELECT=88,keystone:DELETE=10 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:12.528550 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee7787b-3933-4221-9076-e1fa20098ff0 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:12.529147 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee7787b-3933-4221-9076-e1fa20098ff0 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:12.529367 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee7787b-3933-4221-9076-e1fa20098ff0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:12.562720 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-4ee7787b-3933-4221-9076-e1fa20098ff0 None None] MFA Rules not processed for user `2d0ba2ae3ffe4bb9b99f1347719df411`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:12.565211 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-4ee7787b-3933-4221-9076-e1fa20098ff0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:12.576573 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:12.592624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b71e04-5e68-444b-9e99-dc9c0cfd1f43 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:12.597377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d7700d-3f02-428e-845b-44c956c4279f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:12.598114 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d7700d-3f02-428e-845b-44c956c4279f service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:12.598460 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d7700d-3f02-428e-845b-44c956c4279f service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:12.598803 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d7700d-3f02-428e-845b-44c956c4279f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:12.610266 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d7700d-3f02-428e-845b-44c956c4279f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:12.612075 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d7700d-3f02-428e-845b-44c956c4279f service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:12.624851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1081/2173] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:21:12.641319 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1093/2174] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:12 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:12.969904 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d7700d-3f02-428e-845b-44c956c4279f service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:12.971402 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-93d7700d-3f02-428e-845b-44c956c4279f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:12.998392 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d7700d-3f02-428e-845b-44c956c4279f service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:13.004321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e398c7a-35ae-4e53-b701-6d7be9e52e1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:13.005128 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e398c7a-35ae-4e53-b701-6d7be9e52e1e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:13.005475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e398c7a-35ae-4e53-b701-6d7be9e52e1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:13.005898 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e398c7a-35ae-4e53-b701-6d7be9e52e1e admin admin] PATH_INFO: `/v3/projects/ffc59ae7bbce4d56aace17cdd05ecceb` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:13.007083 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e398c7a-35ae-4e53-b701-6d7be9e52e1e admin admin] RBAC: Authorizing `identity:delete_project(project_id=ffc59ae7bbce4d56aace17cdd05ecceb)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:13.009001 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e398c7a-35ae-4e53-b701-6d7be9e52e1e admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:13.068636 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-3e398c7a-35ae-4e53-b701-6d7be9e52e1e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ffc59ae7bbce4d56aace17cdd05ecceb', 'request_id': 'req-3e398c7a-35ae-4e53-b701-6d7be9e52e1e'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:13.074188 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-3e398c7a-35ae-4e53-b701-6d7be9e52e1e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ffc59ae7bbce4d56aace17cdd05ecceb', 'request_id': 'req-3e398c7a-35ae-4e53-b701-6d7be9e52e1e'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:13.080175 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-3e398c7a-35ae-4e53-b701-6d7be9e52e1e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ffc59ae7bbce4d56aace17cdd05ecceb', 'request_id': 'req-3e398c7a-35ae-4e53-b701-6d7be9e52e1e'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:13.091341 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-3e398c7a-35ae-4e53-b701-6d7be9e52e1e admin admin] The token cache is being invalidate because project ffc59ae7bbce4d56aace17cdd05ecceb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:21:13.091623 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-3e398c7a-35ae-4e53-b701-6d7be9e52e1e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:13.092235 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-3e398c7a-35ae-4e53-b701-6d7be9e52e1e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:13.093730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1082/2175] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:21:12 2022] DELETE /identity/v3/projects/ffc59ae7bbce4d56aace17cdd05ecceb => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:13.097717 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee7787b-3933-4221-9076-e1fa20098ff0 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:13.108854 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee7787b-3933-4221-9076-e1fa20098ff0 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:13.170391 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5647ee-0415-42a3-a652-f100bf13a670 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:13.171560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5647ee-0415-42a3-a652-f100bf13a670 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:13.171838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5647ee-0415-42a3-a652-f100bf13a670 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:13.172111 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5647ee-0415-42a3-a652-f100bf13a670 admin admin] PATH_INFO: `/v3/users/525171b955ed4b7480fcc720d0cf4d5c` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:13.173288 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5647ee-0415-42a3-a652-f100bf13a670 admin admin] RBAC: Authorizing `identity:delete_user(user_id=525171b955ed4b7480fcc720d0cf4d5c)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:13.175241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5647ee-0415-42a3-a652-f100bf13a670 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:13.229804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-cb5647ee-0415-42a3-a652-f100bf13a670 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '525171b955ed4b7480fcc720d0cf4d5c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:13.237403 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-cb5647ee-0415-42a3-a652-f100bf13a670 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '525171b955ed4b7480fcc720d0cf4d5c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:13.238313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-cb5647ee-0415-42a3-a652-f100bf13a670 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '525171b955ed4b7480fcc720d0cf4d5c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:13.250670 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-cb5647ee-0415-42a3-a652-f100bf13a670 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '525171b955ed4b7480fcc720d0cf4d5c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:13.276495 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-cb5647ee-0415-42a3-a652-f100bf13a670 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '525171b955ed4b7480fcc720d0cf4d5c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:13.283313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1094/2176] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:21:13 2022] DELETE /identity/v3/users/525171b955ed4b7480fcc720d0cf4d5c => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:13.582704 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e398c7a-35ae-4e53-b701-6d7be9e52e1e admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:13.585950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-3e398c7a-35ae-4e53-b701-6d7be9e52e1e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:13.600982 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e398c7a-35ae-4e53-b701-6d7be9e52e1e admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:13.688110 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f24f3b-d006-4c37-a164-34d8f964943d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:13.689329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f24f3b-d006-4c37-a164-34d8f964943d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:13.689864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f24f3b-d006-4c37-a164-34d8f964943d admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:13.690284 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f24f3b-d006-4c37-a164-34d8f964943d admin admin] PATH_INFO: `/v3/projects/91f011314af940eda4e4fdb66a3c8226` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:13.691742 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f24f3b-d006-4c37-a164-34d8f964943d admin admin] RBAC: Authorizing `identity:delete_project(project_id=91f011314af940eda4e4fdb66a3c8226)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:13.693710 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f24f3b-d006-4c37-a164-34d8f964943d admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:13.738659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:13.760722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-24f24f3b-d006-4c37-a164-34d8f964943d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '91f011314af940eda4e4fdb66a3c8226', 'request_id': 'req-24f24f3b-d006-4c37-a164-34d8f964943d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:13.767297 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-24f24f3b-d006-4c37-a164-34d8f964943d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '91f011314af940eda4e4fdb66a3c8226', 'request_id': 'req-24f24f3b-d006-4c37-a164-34d8f964943d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:13.786997 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-24f24f3b-d006-4c37-a164-34d8f964943d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '91f011314af940eda4e4fdb66a3c8226', 'request_id': 'req-24f24f3b-d006-4c37-a164-34d8f964943d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:13.802880 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-24f24f3b-d006-4c37-a164-34d8f964943d admin admin] The token cache is being invalidate because project 91f011314af940eda4e4fdb66a3c8226 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:21:13.803259 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-24f24f3b-d006-4c37-a164-34d8f964943d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:13.804002 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-24f24f3b-d006-4c37-a164-34d8f964943d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:13.805602 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1083/2177] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:21:13 2022] DELETE /identity/v3/projects/91f011314af940eda4e4fdb66a3c8226 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:13.812192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc94b2cd-bdf2-40b8-a73d-f73f654edf8f None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:13.812686 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc94b2cd-bdf2-40b8-a73d-f73f654edf8f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:13.812852 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc94b2cd-bdf2-40b8-a73d-f73f654edf8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:13.843776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-fc94b2cd-bdf2-40b8-a73d-f73f654edf8f None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:13.845036 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-fc94b2cd-bdf2-40b8-a73d-f73f654edf8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:13.875505 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:13.908326 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1095/2178] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:13.917296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1975a1-bb2b-45b6-a4b9-493e3173e23e None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:13.918065 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1975a1-bb2b-45b6-a4b9-493e3173e23e None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:13.918467 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1975a1-bb2b-45b6-a4b9-493e3173e23e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:13.942135 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-7f1975a1-bb2b-45b6-a4b9-493e3173e23e None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:13.943375 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-7f1975a1-bb2b-45b6-a4b9-493e3173e23e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:13.952549 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1084/2179] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:13.956431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc94b2cd-bdf2-40b8-a73d-f73f654edf8f None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:13.966134 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc94b2cd-bdf2-40b8-a73d-f73f654edf8f None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:13.970660 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebd6733-e910-4a06-86c1-1bc906101409 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:13.971245 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebd6733-e910-4a06-86c1-1bc906101409 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:13.971518 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebd6733-e910-4a06-86c1-1bc906101409 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:13.971786 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebd6733-e910-4a06-86c1-1bc906101409 admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:13.972423 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cebd6733-e910-4a06-86c1-1bc906101409 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:21:13.972714 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cebd6733-e910-4a06-86c1-1bc906101409 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:13.974311 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cebd6733-e910-4a06-86c1-1bc906101409 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:13.986561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1096/2180] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:21:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:21:13.991311 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1975a1-bb2b-45b6-a4b9-493e3173e23e None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:14.001330 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1975a1-bb2b-45b6-a4b9-493e3173e23e None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:14.005973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad3f4a7-08e0-4f69-bb90-32ac915d1073 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:14.006579 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad3f4a7-08e0-4f69-bb90-32ac915d1073 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:14.006856 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad3f4a7-08e0-4f69-bb90-32ac915d1073 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:14.007126 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad3f4a7-08e0-4f69-bb90-32ac915d1073 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:14.007754 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ad3f4a7-08e0-4f69-bb90-32ac915d1073 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:14.009635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ad3f4a7-08e0-4f69-bb90-32ac915d1073 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:14.043398 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1085/2181] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:21:13 2022] POST /identity/v3/projects => generated 361 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:21:14.044634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebd6733-e910-4a06-86c1-1bc906101409 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:14.058443 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebd6733-e910-4a06-86c1-1bc906101409 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:14.129221 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6a257a-f4b0-490a-abbd-ee85342ad072 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:14.129982 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6a257a-f4b0-490a-abbd-ee85342ad072 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:14.130337 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6a257a-f4b0-490a-abbd-ee85342ad072 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:14.130784 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6a257a-f4b0-490a-abbd-ee85342ad072 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:14.131478 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6a257a-f4b0-490a-abbd-ee85342ad072 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:14.133456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6a257a-f4b0-490a-abbd-ee85342ad072 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:14.162979 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1097/2182] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:21:14 2022] POST /identity/v3/users => generated 281 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:21:14.167356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad3f4a7-08e0-4f69-bb90-32ac915d1073 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:14.178213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad3f4a7-08e0-4f69-bb90-32ac915d1073 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:14.183386 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcac37a1-6827-4df9-95aa-65ce9831b628 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:14.184091 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcac37a1-6827-4df9-95aa-65ce9831b628 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:14.184369 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcac37a1-6827-4df9-95aa-65ce9831b628 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:14.184639 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcac37a1-6827-4df9-95aa-65ce9831b628 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:14.185286 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcac37a1-6827-4df9-95aa-65ce9831b628 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:14.186947 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcac37a1-6827-4df9-95aa-65ce9831b628 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:14.199846 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1086/2183] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:21:14.201389 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6a257a-f4b0-490a-abbd-ee85342ad072 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:14.211247 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6a257a-f4b0-490a-abbd-ee85342ad072 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:14.215863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7570758-61f8-4e56-b780-731b4e0e38f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:14.216574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7570758-61f8-4e56-b780-731b4e0e38f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:14.216863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7570758-61f8-4e56-b780-731b4e0e38f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:14.217198 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7570758-61f8-4e56-b780-731b4e0e38f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:14.218464 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7570758-61f8-4e56-b780-731b4e0e38f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:14.220102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7570758-61f8-4e56-b780-731b4e0e38f0 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:14.230775 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1098/2184] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:21:14.235669 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcac37a1-6827-4df9-95aa-65ce9831b628 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:14.245696 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcac37a1-6827-4df9-95aa-65ce9831b628 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:14.251217 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7205c7-31b3-430d-b560-4054b5a2eea1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:14.251884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7205c7-31b3-430d-b560-4054b5a2eea1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:14.252170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7205c7-31b3-430d-b560-4054b5a2eea1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:14.252446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7205c7-31b3-430d-b560-4054b5a2eea1 admin admin] PATH_INFO: `/v3/projects/7b1a405bde9742b5acdcb188057441ae/users/f976615388bb46c7be6b035b755f1d59/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:14.269818 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7205c7-31b3-430d-b560-4054b5a2eea1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b1a405bde9742b5acdcb188057441ae, user_id=f976615388bb46c7be6b035b755f1d59, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:14.272466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7205c7-31b3-430d-b560-4054b5a2eea1 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:14.287880 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1087/2185] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:21:14 2022] PUT /identity/v3/projects/7b1a405bde9742b5acdcb188057441ae/users/f976615388bb46c7be6b035b755f1d59/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:14.299286 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67435f8-6ae3-451e-a5a4-d633cdaa6f76 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:14.299570 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67435f8-6ae3-451e-a5a4-d633cdaa6f76 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:14.299843 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67435f8-6ae3-451e-a5a4-d633cdaa6f76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:14.338947 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-b67435f8-6ae3-451e-a5a4-d633cdaa6f76 None None] MFA Rules not processed for user `f976615388bb46c7be6b035b755f1d59`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:14.340388 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-b67435f8-6ae3-451e-a5a4-d633cdaa6f76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:14.426025 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1099/2186] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:14 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:14.464711 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7205c7-31b3-430d-b560-4054b5a2eea1 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:14.466665 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-5d7205c7-31b3-430d-b560-4054b5a2eea1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:14.540862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7205c7-31b3-430d-b560-4054b5a2eea1 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:14.544659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f542abc-9dbf-4b55-bebe-492fc24486d2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:14.545264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f542abc-9dbf-4b55-bebe-492fc24486d2 service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:14.545537 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f542abc-9dbf-4b55-bebe-492fc24486d2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:14.546394 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f542abc-9dbf-4b55-bebe-492fc24486d2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:14.554868 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f542abc-9dbf-4b55-bebe-492fc24486d2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:14.556284 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f542abc-9dbf-4b55-bebe-492fc24486d2 service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:14.565952 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1088/2187] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:14 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:21:14.581486 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67435f8-6ae3-451e-a5a4-d633cdaa6f76 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:14.615465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:14.658255 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67435f8-6ae3-451e-a5a4-d633cdaa6f76 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:14.663139 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63e43e4-6f8d-4bc9-82f0-50cd1f3c2523 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:14.663776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63e43e4-6f8d-4bc9-82f0-50cd1f3c2523 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:14.664065 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63e43e4-6f8d-4bc9-82f0-50cd1f3c2523 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:14.664323 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63e43e4-6f8d-4bc9-82f0-50cd1f3c2523 admin admin] PATH_INFO: `/v3/users/f976615388bb46c7be6b035b755f1d59` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:14.665447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f63e43e4-6f8d-4bc9-82f0-50cd1f3c2523 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f976615388bb46c7be6b035b755f1d59)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:14.667526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f63e43e4-6f8d-4bc9-82f0-50cd1f3c2523 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:14.725726 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-f63e43e4-6f8d-4bc9-82f0-50cd1f3c2523 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f976615388bb46c7be6b035b755f1d59'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:14.733842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-f63e43e4-6f8d-4bc9-82f0-50cd1f3c2523 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f976615388bb46c7be6b035b755f1d59'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:14.734600 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-f63e43e4-6f8d-4bc9-82f0-50cd1f3c2523 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f976615388bb46c7be6b035b755f1d59'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:14.744125 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-f63e43e4-6f8d-4bc9-82f0-50cd1f3c2523 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f976615388bb46c7be6b035b755f1d59'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:14.763733 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-f63e43e4-6f8d-4bc9-82f0-50cd1f3c2523 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f976615388bb46c7be6b035b755f1d59'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:14.771815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1100/2188] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:21:14 2022] DELETE /identity/v3/users/f976615388bb46c7be6b035b755f1d59 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:21:14.776326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf94e9f-15b4-4bd7-854e-259f5269c9ac None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:14.776811 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf94e9f-15b4-4bd7-854e-259f5269c9ac None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:14.777015 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf94e9f-15b4-4bd7-854e-259f5269c9ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:14.797279 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-9bf94e9f-15b4-4bd7-854e-259f5269c9ac None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:14.798507 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-9bf94e9f-15b4-4bd7-854e-259f5269c9ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:14.871560 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1089/2189] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:21:14.881259 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63e43e4-6f8d-4bc9-82f0-50cd1f3c2523 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:14.882765 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-f63e43e4-6f8d-4bc9-82f0-50cd1f3c2523 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:14.940666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63e43e4-6f8d-4bc9-82f0-50cd1f3c2523 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:14.944307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a9daf3-73ff-40f8-8030-796a56c10eb2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:14.944917 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a9daf3-73ff-40f8-8030-796a56c10eb2 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:14.945195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a9daf3-73ff-40f8-8030-796a56c10eb2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:14.945466 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a9daf3-73ff-40f8-8030-796a56c10eb2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:14.954552 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a9daf3-73ff-40f8-8030-796a56c10eb2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:14.956012 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a9daf3-73ff-40f8-8030-796a56c10eb2 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:14.965486 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1101/2190] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:21:15.280815 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf94e9f-15b4-4bd7-854e-259f5269c9ac None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:15.282881 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-9bf94e9f-15b4-4bd7-854e-259f5269c9ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:15.293484 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:15.301018 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf94e9f-15b4-4bd7-854e-259f5269c9ac None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:15.311089 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055c9835-95d8-4320-b40e-4112cdf33e42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:15.320238 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055c9835-95d8-4320-b40e-4112cdf33e42 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:15.320672 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055c9835-95d8-4320-b40e-4112cdf33e42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:15.321045 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055c9835-95d8-4320-b40e-4112cdf33e42 admin admin] PATH_INFO: `/v3/projects/7b1a405bde9742b5acdcb188057441ae` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:15.322531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055c9835-95d8-4320-b40e-4112cdf33e42 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7b1a405bde9742b5acdcb188057441ae)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:15.331866 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055c9835-95d8-4320-b40e-4112cdf33e42 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:15.410851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-055c9835-95d8-4320-b40e-4112cdf33e42 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7b1a405bde9742b5acdcb188057441ae', 'request_id': 'req-055c9835-95d8-4320-b40e-4112cdf33e42'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:15.418025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-055c9835-95d8-4320-b40e-4112cdf33e42 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7b1a405bde9742b5acdcb188057441ae', 'request_id': 'req-055c9835-95d8-4320-b40e-4112cdf33e42'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:15.427581 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-055c9835-95d8-4320-b40e-4112cdf33e42 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7b1a405bde9742b5acdcb188057441ae', 'request_id': 'req-055c9835-95d8-4320-b40e-4112cdf33e42'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:15.441835 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-055c9835-95d8-4320-b40e-4112cdf33e42 admin admin] The token cache is being invalidate because project 7b1a405bde9742b5acdcb188057441ae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:21:15.442169 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-055c9835-95d8-4320-b40e-4112cdf33e42 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:15.442860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-055c9835-95d8-4320-b40e-4112cdf33e42 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:15.444542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1090/2191] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:21:15 2022] DELETE /identity/v3/projects/7b1a405bde9742b5acdcb188057441ae => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:21:15.451642 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b17e35-ab14-4a22-8219-773b5f4c0c7b None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:15.452752 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b17e35-ab14-4a22-8219-773b5f4c0c7b None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:15.453210 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b17e35-ab14-4a22-8219-773b5f4c0c7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:15.479077 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-f2b17e35-ab14-4a22-8219-773b5f4c0c7b None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:15.480619 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-f2b17e35-ab14-4a22-8219-773b5f4c0c7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:15.556360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:15.585857 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1102/2192] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:15.594267 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de067032-27e8-42db-aa2d-7c60d2a6eb06 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:15.594852 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de067032-27e8-42db-aa2d-7c60d2a6eb06 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:15.595199 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de067032-27e8-42db-aa2d-7c60d2a6eb06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:15.618654 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-de067032-27e8-42db-aa2d-7c60d2a6eb06 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:15.620022 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-de067032-27e8-42db-aa2d-7c60d2a6eb06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:15.630217 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1091/2193] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:15.634646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b17e35-ab14-4a22-8219-773b5f4c0c7b None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:15.645491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b17e35-ab14-4a22-8219-773b5f4c0c7b None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:15.650681 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d9f6d7-8b29-49ce-8a5f-db34476ca060 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:15.651539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d9f6d7-8b29-49ce-8a5f-db34476ca060 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:15.652081 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d9f6d7-8b29-49ce-8a5f-db34476ca060 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:15.652567 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d9f6d7-8b29-49ce-8a5f-db34476ca060 admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:15.653476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d9f6d7-8b29-49ce-8a5f-db34476ca060 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:21:15.654444 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d9f6d7-8b29-49ce-8a5f-db34476ca060 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:15.656724 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d9f6d7-8b29-49ce-8a5f-db34476ca060 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:15.676415 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1103/2194] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:21:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:21:15.677657 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de067032-27e8-42db-aa2d-7c60d2a6eb06 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:15.687914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de067032-27e8-42db-aa2d-7c60d2a6eb06 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:15.692736 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea1b84d-e357-42de-a309-7948adc9bb5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:15.693417 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea1b84d-e357-42de-a309-7948adc9bb5b admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:15.693795 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea1b84d-e357-42de-a309-7948adc9bb5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:15.694469 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea1b84d-e357-42de-a309-7948adc9bb5b admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:15.694784 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea1b84d-e357-42de-a309-7948adc9bb5b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:15.696670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea1b84d-e357-42de-a309-7948adc9bb5b admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:15.730238 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1092/2195] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:21:15 2022] POST /identity/v3/projects => generated 385 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:21:15.734074 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d9f6d7-8b29-49ce-8a5f-db34476ca060 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:15.744936 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d9f6d7-8b29-49ce-8a5f-db34476ca060 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:15.834327 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54716256-fb43-4697-95a4-206045099cb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:15.835187 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54716256-fb43-4697-95a4-206045099cb5 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:15.835482 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54716256-fb43-4697-95a4-206045099cb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:15.835758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54716256-fb43-4697-95a4-206045099cb5 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:15.836414 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54716256-fb43-4697-95a4-206045099cb5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:15.838655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54716256-fb43-4697-95a4-206045099cb5 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:15.869017 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1104/2196] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:21:15 2022] POST /identity/v3/users => generated 293 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:21:15.875422 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea1b84d-e357-42de-a309-7948adc9bb5b admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:15.894977 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea1b84d-e357-42de-a309-7948adc9bb5b admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:15.910454 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba09858-7e75-4ee3-93db-4dec19fa0b1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:15.911364 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba09858-7e75-4ee3-93db-4dec19fa0b1a admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:15.912003 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba09858-7e75-4ee3-93db-4dec19fa0b1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:15.912700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba09858-7e75-4ee3-93db-4dec19fa0b1a admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:15.913552 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba09858-7e75-4ee3-93db-4dec19fa0b1a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:15.915490 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba09858-7e75-4ee3-93db-4dec19fa0b1a admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:15.930983 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1093/2197] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:15 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:21:15.938913 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54716256-fb43-4697-95a4-206045099cb5 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:15.954442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54716256-fb43-4697-95a4-206045099cb5 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:15.959791 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d1f7f1-59fb-4948-bfb2-a0bf89251e25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:15.960403 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d1f7f1-59fb-4948-bfb2-a0bf89251e25 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:15.960681 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d1f7f1-59fb-4948-bfb2-a0bf89251e25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:15.960959 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d1f7f1-59fb-4948-bfb2-a0bf89251e25 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:15.961606 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42d1f7f1-59fb-4948-bfb2-a0bf89251e25 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:15.963401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42d1f7f1-59fb-4948-bfb2-a0bf89251e25 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:15.976621 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1105/2198] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:15 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:21:15.981066 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba09858-7e75-4ee3-93db-4dec19fa0b1a admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:15.992825 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba09858-7e75-4ee3-93db-4dec19fa0b1a admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:16.009120 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae6b441-3204-4fdb-abd5-4182725d0639 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:16.009973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae6b441-3204-4fdb-abd5-4182725d0639 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:16.010129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae6b441-3204-4fdb-abd5-4182725d0639 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:16.010735 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae6b441-3204-4fdb-abd5-4182725d0639 admin admin] PATH_INFO: `/v3/projects/7e7b2ff0f0104769a550d0c0692ab713/users/42d91d56296e4fdd917495df41401921/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:16.026784 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae6b441-3204-4fdb-abd5-4182725d0639 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7e7b2ff0f0104769a550d0c0692ab713, user_id=42d91d56296e4fdd917495df41401921, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:16.029790 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae6b441-3204-4fdb-abd5-4182725d0639 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:16.048069 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1094/2199] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:21:15 2022] PUT /identity/v3/projects/7e7b2ff0f0104769a550d0c0692ab713/users/42d91d56296e4fdd917495df41401921/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:16.053406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704773e9-9632-460a-8a7f-904d8b24aef8 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:16.053687 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704773e9-9632-460a-8a7f-904d8b24aef8 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:16.054014 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704773e9-9632-460a-8a7f-904d8b24aef8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:16.075474 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-704773e9-9632-460a-8a7f-904d8b24aef8 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:16.077132 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-704773e9-9632-460a-8a7f-904d8b24aef8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:16.173796 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1106/2200] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:21:16.182717 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae6b441-3204-4fdb-abd5-4182725d0639 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:16.184728 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-aae6b441-3204-4fdb-abd5-4182725d0639 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:16.251176 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae6b441-3204-4fdb-abd5-4182725d0639 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:16.255191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1935cf-05e3-4aba-9877-5cb3710f5565 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:16.255797 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1935cf-05e3-4aba-9877-5cb3710f5565 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:16.256072 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1935cf-05e3-4aba-9877-5cb3710f5565 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:16.256343 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1935cf-05e3-4aba-9877-5cb3710f5565 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:16.265848 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1935cf-05e3-4aba-9877-5cb3710f5565 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:16.267242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1935cf-05e3-4aba-9877-5cb3710f5565 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:16.276595 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1095/2201] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:21:21.489518 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe2a044-6dcf-4311-8358-24f07f7d8670 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:21.490049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe2a044-6dcf-4311-8358-24f07f7d8670 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:21.490131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe2a044-6dcf-4311-8358-24f07f7d8670 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:21.523822 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-cbe2a044-6dcf-4311-8358-24f07f7d8670 None None] MFA Rules not processed for user `42d91d56296e4fdd917495df41401921`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:21.525287 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-cbe2a044-6dcf-4311-8358-24f07f7d8670 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:21.603119 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:21.623023 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1107/2202] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:21 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:21.628723 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1935cf-05e3-4aba-9877-5cb3710f5565 service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:21.638551 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1935cf-05e3-4aba-9877-5cb3710f5565 service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:21.644290 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaec2e91-f236-4fcf-875c-23575ae36e8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:21.645319 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaec2e91-f236-4fcf-875c-23575ae36e8b admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:21.645604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaec2e91-f236-4fcf-875c-23575ae36e8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:21.645933 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaec2e91-f236-4fcf-875c-23575ae36e8b admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:21.646605 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaec2e91-f236-4fcf-875c-23575ae36e8b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:21.648496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaec2e91-f236-4fcf-875c-23575ae36e8b admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:21.678871 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1096/2203] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:21:21 2022] POST /identity/v3/projects => generated 385 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:21:21.683935 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe2a044-6dcf-4311-8358-24f07f7d8670 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:21.710370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe2a044-6dcf-4311-8358-24f07f7d8670 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:21.795399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2c2f99-7159-41fb-a5e4-f0b0fcea3c90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:21.796039 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2c2f99-7159-41fb-a5e4-f0b0fcea3c90 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:21.796316 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2c2f99-7159-41fb-a5e4-f0b0fcea3c90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:21.796587 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2c2f99-7159-41fb-a5e4-f0b0fcea3c90 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:21.797218 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2c2f99-7159-41fb-a5e4-f0b0fcea3c90 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:21.801272 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2c2f99-7159-41fb-a5e4-f0b0fcea3c90 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:21.835096 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1108/2204] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:21:21 2022] POST /identity/v3/users => generated 299 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:21:21.851699 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaec2e91-f236-4fcf-875c-23575ae36e8b admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:21.860811 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:21.865007 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaec2e91-f236-4fcf-875c-23575ae36e8b admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:21.871713 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b9167b-61b0-4fe9-b86a-30e503e94816 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:21.872475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b9167b-61b0-4fe9-b86a-30e503e94816 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:21.872917 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b9167b-61b0-4fe9-b86a-30e503e94816 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:21.873483 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b9167b-61b0-4fe9-b86a-30e503e94816 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:21.874840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b9167b-61b0-4fe9-b86a-30e503e94816 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:21.876580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b9167b-61b0-4fe9-b86a-30e503e94816 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:21.891389 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1097/2205] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:21 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:21:21.895620 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2c2f99-7159-41fb-a5e4-f0b0fcea3c90 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:21.907466 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2c2f99-7159-41fb-a5e4-f0b0fcea3c90 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:21.913899 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c229a793-237e-4457-a348-885f8fee8a5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:21.914603 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c229a793-237e-4457-a348-885f8fee8a5c admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:21.914907 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c229a793-237e-4457-a348-885f8fee8a5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:21.915181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c229a793-237e-4457-a348-885f8fee8a5c admin admin] PATH_INFO: `/v3/projects/3030064c1315435699a7e7d5a831a5b2/users/01c2a355a3e8409a8bd7340cf5979036/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:21.932485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c229a793-237e-4457-a348-885f8fee8a5c admin admin] RBAC: Authorizing `identity:create_grant(project_id=3030064c1315435699a7e7d5a831a5b2, user_id=01c2a355a3e8409a8bd7340cf5979036, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:21.935281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c229a793-237e-4457-a348-885f8fee8a5c admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:21.949668 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1109/2206] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:21:21 2022] PUT /identity/v3/projects/3030064c1315435699a7e7d5a831a5b2/users/01c2a355a3e8409a8bd7340cf5979036/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:21.953893 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b9167b-61b0-4fe9-b86a-30e503e94816 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:21.965591 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b9167b-61b0-4fe9-b86a-30e503e94816 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:22.028344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36954dfa-96a0-4029-a476-8eb2eb967d3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:22.028961 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36954dfa-96a0-4029-a476-8eb2eb967d3d admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:22.029512 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36954dfa-96a0-4029-a476-8eb2eb967d3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:22.029818 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36954dfa-96a0-4029-a476-8eb2eb967d3d admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:22.030514 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36954dfa-96a0-4029-a476-8eb2eb967d3d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:22.032277 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36954dfa-96a0-4029-a476-8eb2eb967d3d admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:22.042775 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1098/2207] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:21 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:21:22.047469 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c229a793-237e-4457-a348-885f8fee8a5c admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:22.059505 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c229a793-237e-4457-a348-885f8fee8a5c admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:22.065228 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bfe2e0-8b61-42c3-86dc-e15f9d86a8ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:22.067765 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bfe2e0-8b61-42c3-86dc-e15f9d86a8ef admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:22.068048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bfe2e0-8b61-42c3-86dc-e15f9d86a8ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:22.068326 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bfe2e0-8b61-42c3-86dc-e15f9d86a8ef admin admin] PATH_INFO: `/v3/projects/3030064c1315435699a7e7d5a831a5b2/users/01c2a355a3e8409a8bd7340cf5979036/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:22.070451 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6bfe2e0-8b61-42c3-86dc-e15f9d86a8ef admin admin] RBAC: Authorizing `identity:create_grant(project_id=3030064c1315435699a7e7d5a831a5b2, user_id=01c2a355a3e8409a8bd7340cf5979036, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:22.073112 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6bfe2e0-8b61-42c3-86dc-e15f9d86a8ef admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:22.084678 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1110/2208] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:21:22 2022] PUT /identity/v3/projects/3030064c1315435699a7e7d5a831a5b2/users/01c2a355a3e8409a8bd7340cf5979036/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:27.160523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88411d64-90f2-4c74-a7c3-675e204186e5 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:27.161103 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88411d64-90f2-4c74-a7c3-675e204186e5 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:27.161103 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88411d64-90f2-4c74-a7c3-675e204186e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:27.190560 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-88411d64-90f2-4c74-a7c3-675e204186e5 None None] MFA Rules not processed for user `01c2a355a3e8409a8bd7340cf5979036`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:27.191702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-88411d64-90f2-4c74-a7c3-675e204186e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:27.284137 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1099/2209] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:27 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:27.297000 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bfe2e0-8b61-42c3-86dc-e15f9d86a8ef admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:27.298954 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a6bfe2e0-8b61-42c3-86dc-e15f9d86a8ef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:27.366251 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bfe2e0-8b61-42c3-86dc-e15f9d86a8ef admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:27.372425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48d5446-bac5-4e24-a658-f9d141f420cd service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dcc53332e6b462baf349a72b19d2b3b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:27.373135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48d5446-bac5-4e24-a658-f9d141f420cd service glance] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:27.373648 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48d5446-bac5-4e24-a658-f9d141f420cd service glance] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:27.374050 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48d5446-bac5-4e24-a658-f9d141f420cd service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:27.451491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:27.456069 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48d5446-bac5-4e24-a658-f9d141f420cd service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:27.458198 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48d5446-bac5-4e24-a658-f9d141f420cd service glance] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:27.470652 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1111/2210] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:27 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:21:27.681925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5596d8f6-128a-4682-9758-8262cd3a1719 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:27.682219 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5596d8f6-128a-4682-9758-8262cd3a1719 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:27.682501 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5596d8f6-128a-4682-9758-8262cd3a1719 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:27.716922 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-5596d8f6-128a-4682-9758-8262cd3a1719 None None] MFA Rules not processed for user `5a2fcc8c9ac44dceadcc30d163bbfa0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:27.718504 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-5596d8f6-128a-4682-9758-8262cd3a1719 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:27.836375 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1100/2211] 10.222.0.26 () {58 vars in 1009 bytes} [Tue Sep 27 04:21:27 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:27.891454 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48d5446-bac5-4e24-a658-f9d141f420cd service glance] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:27.893410 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-b48d5446-bac5-4e24-a658-f9d141f420cd service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:28.000523 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48d5446-bac5-4e24-a658-f9d141f420cd service glance] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:28.009102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8bf927-7492-49f8-b4fc-d8f82c354cc8 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '181dd5dc720c4a97860852062b7b19c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:28.009736 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8bf927-7492-49f8-b4fc-d8f82c354cc8 service swift] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:28.010347 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8bf927-7492-49f8-b4fc-d8f82c354cc8 service swift] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:28.010509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8bf927-7492-49f8-b4fc-d8f82c354cc8 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:28.023187 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8bf927-7492-49f8-b4fc-d8f82c354cc8 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:28.030027 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8bf927-7492-49f8-b4fc-d8f82c354cc8 service swift] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:28.054268 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1112/2212] 10.222.0.26 () {60 vars in 1384 bytes} [Tue Sep 27 04:21:27 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 04:21:31.436349 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5596d8f6-128a-4682-9758-8262cd3a1719 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:31.438309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-5596d8f6-128a-4682-9758-8262cd3a1719 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:31.488366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:31.569372 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5596d8f6-128a-4682-9758-8262cd3a1719 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:31.588399 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8325260-980e-48b5-a83e-2dae34c364c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:31.589071 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8325260-980e-48b5-a83e-2dae34c364c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:31.590215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8325260-980e-48b5-a83e-2dae34c364c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:31.590215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8325260-980e-48b5-a83e-2dae34c364c3 admin admin] PATH_INFO: `/v3/users/ac275ebbabeb46018627ff0259450acf` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:31.591550 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8325260-980e-48b5-a83e-2dae34c364c3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ac275ebbabeb46018627ff0259450acf)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:31.594940 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8325260-980e-48b5-a83e-2dae34c364c3 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:31.713833 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-b8325260-980e-48b5-a83e-2dae34c364c3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ac275ebbabeb46018627ff0259450acf'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:31.738402 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-b8325260-980e-48b5-a83e-2dae34c364c3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ac275ebbabeb46018627ff0259450acf'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:31.739318 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-b8325260-980e-48b5-a83e-2dae34c364c3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ac275ebbabeb46018627ff0259450acf'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:31.759044 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-b8325260-980e-48b5-a83e-2dae34c364c3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ac275ebbabeb46018627ff0259450acf'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:31.786713 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-b8325260-980e-48b5-a83e-2dae34c364c3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ac275ebbabeb46018627ff0259450acf'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:31.804406 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1101/2213] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:21:31 2022] DELETE /identity/v3/users/ac275ebbabeb46018627ff0259450acf => generated 0 bytes in 365 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:31.805767 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132b02e3-62bf-458c-a683-238b2ab6988d None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:31.806721 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132b02e3-62bf-458c-a683-238b2ab6988d None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:31.807233 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132b02e3-62bf-458c-a683-238b2ab6988d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:31.862247 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-132b02e3-62bf-458c-a683-238b2ab6988d None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:31.864209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-132b02e3-62bf-458c-a683-238b2ab6988d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:32.056395 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1113/2214] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 252 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:21:32.065729 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8325260-980e-48b5-a83e-2dae34c364c3 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:32.069423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-b8325260-980e-48b5-a83e-2dae34c364c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:32.212235 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8325260-980e-48b5-a83e-2dae34c364c3 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:32.218508 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad36fcb8-d911-46cc-a1b1-01d2b34326db service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:32.219310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad36fcb8-d911-46cc-a1b1-01d2b34326db service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:32.219805 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad36fcb8-d911-46cc-a1b1-01d2b34326db service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:32.220226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad36fcb8-d911-46cc-a1b1-01d2b34326db service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:32.238714 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad36fcb8-d911-46cc-a1b1-01d2b34326db service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:32.241719 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad36fcb8-d911-46cc-a1b1-01d2b34326db service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:32.276138 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1102/2215] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:21:32.646689 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132b02e3-62bf-458c-a683-238b2ab6988d None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:32.648539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-132b02e3-62bf-458c-a683-238b2ab6988d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:32.664413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132b02e3-62bf-458c-a683-238b2ab6988d None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:32.676413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc56c9a-9a75-455c-8797-031e074d51ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:32.677075 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc56c9a-9a75-455c-8797-031e074d51ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:32.677353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc56c9a-9a75-455c-8797-031e074d51ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:32.677862 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc56c9a-9a75-455c-8797-031e074d51ed admin admin] PATH_INFO: `/v3/users/2e505b67d0064ff2a255c2d4cc47248c` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:32.679580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc56c9a-9a75-455c-8797-031e074d51ed admin admin] RBAC: Authorizing `identity:delete_user(user_id=2e505b67d0064ff2a255c2d4cc47248c)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:32.681507 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc56c9a-9a75-455c-8797-031e074d51ed admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:32.834386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-5cc56c9a-9a75-455c-8797-031e074d51ed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2e505b67d0064ff2a255c2d4cc47248c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:32.864779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-5cc56c9a-9a75-455c-8797-031e074d51ed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2e505b67d0064ff2a255c2d4cc47248c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:32.869209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-5cc56c9a-9a75-455c-8797-031e074d51ed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2e505b67d0064ff2a255c2d4cc47248c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:32.881686 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-5cc56c9a-9a75-455c-8797-031e074d51ed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2e505b67d0064ff2a255c2d4cc47248c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:32.909157 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:32.911069 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-5cc56c9a-9a75-455c-8797-031e074d51ed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2e505b67d0064ff2a255c2d4cc47248c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:32.916643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad36fcb8-d911-46cc-a1b1-01d2b34326db service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:32.918946 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ad36fcb8-d911-46cc-a1b1-01d2b34326db service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:32.930435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1114/2216] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:21:32 2022] DELETE /identity/v3/users/2e505b67d0064ff2a255c2d4cc47248c => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:32.932120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e7ba11-579e-4275-a4cf-10c40bde426f None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:32.941480 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e7ba11-579e-4275-a4cf-10c40bde426f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:32.941853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e7ba11-579e-4275-a4cf-10c40bde426f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:32.950380 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad36fcb8-d911-46cc-a1b1-01d2b34326db service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:32.965340 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-33e7ba11-579e-4275-a4cf-10c40bde426f None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:32.967056 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-33e7ba11-579e-4275-a4cf-10c40bde426f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:33.075230 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:33.100002 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda46778-a3c6-45fe-a318-9c57b765da38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:33.100665 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda46778-a3c6-45fe-a318-9c57b765da38 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:33.100947 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda46778-a3c6-45fe-a318-9c57b765da38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:33.101229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda46778-a3c6-45fe-a318-9c57b765da38 admin admin] PATH_INFO: `/v3/projects/a0c3265513614f2897b64a76b93b7e87` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:33.105629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda46778-a3c6-45fe-a318-9c57b765da38 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a0c3265513614f2897b64a76b93b7e87)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:33.106712 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda46778-a3c6-45fe-a318-9c57b765da38 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:33.153350 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1115/2217] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:33.166641 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e7ba11-579e-4275-a4cf-10c40bde426f None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:33.169228 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-33e7ba11-579e-4275-a4cf-10c40bde426f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:33.238461 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-fda46778-a3c6-45fe-a318-9c57b765da38 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a0c3265513614f2897b64a76b93b7e87', 'request_id': 'req-fda46778-a3c6-45fe-a318-9c57b765da38'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:33.264257 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-fda46778-a3c6-45fe-a318-9c57b765da38 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a0c3265513614f2897b64a76b93b7e87', 'request_id': 'req-fda46778-a3c6-45fe-a318-9c57b765da38'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:33.281356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-fda46778-a3c6-45fe-a318-9c57b765da38 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a0c3265513614f2897b64a76b93b7e87', 'request_id': 'req-fda46778-a3c6-45fe-a318-9c57b765da38'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:33.310359 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-fda46778-a3c6-45fe-a318-9c57b765da38 admin admin] The token cache is being invalidate because project a0c3265513614f2897b64a76b93b7e87 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:21:33.310654 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-fda46778-a3c6-45fe-a318-9c57b765da38 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:33.312300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-fda46778-a3c6-45fe-a318-9c57b765da38 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:33.315804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e7ba11-579e-4275-a4cf-10c40bde426f None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:33.316366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1103/2218] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:21:32 2022] DELETE /identity/v3/projects/a0c3265513614f2897b64a76b93b7e87 => generated 0 bytes in 400 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:33.321196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b693b0c-5145-444d-aa2f-b753cbb4f5a7 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:33.322753 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b693b0c-5145-444d-aa2f-b753cbb4f5a7 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:33.323060 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b693b0c-5145-444d-aa2f-b753cbb4f5a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:33.380864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-6b693b0c-5145-444d-aa2f-b753cbb4f5a7 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:33.385069 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-6b693b0c-5145-444d-aa2f-b753cbb4f5a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:33.465093 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b690a6b-65dd-4d1f-ac25-6ba024129564 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:33.466244 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b690a6b-65dd-4d1f-ac25-6ba024129564 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:33.466525 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b690a6b-65dd-4d1f-ac25-6ba024129564 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:33.466803 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b690a6b-65dd-4d1f-ac25-6ba024129564 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:33.475698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-7b690a6b-65dd-4d1f-ac25-6ba024129564 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:33.568658 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1104/2219] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:21:33.570188 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fef0bf-4d31-471a-b82e-06d8dc858198 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:33.570477 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fef0bf-4d31-471a-b82e-06d8dc858198 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:33.570917 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fef0bf-4d31-471a-b82e-06d8dc858198 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:33.604158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-87fef0bf-4d31-471a-b82e-06d8dc858198 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:33.611007 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-87fef0bf-4d31-471a-b82e-06d8dc858198 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:33.620737 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b690a6b-65dd-4d1f-ac25-6ba024129564 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:33.622378 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b690a6b-65dd-4d1f-ac25-6ba024129564 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:33.642889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1105/2220] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:33.644434 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fef0bf-4d31-471a-b82e-06d8dc858198 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:33.663587 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fef0bf-4d31-471a-b82e-06d8dc858198 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:33.665921 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1116/2221] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 501 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:21:33.685524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6726baf0-67cd-4468-8323-4957261063c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:33.686199 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6726baf0-67cd-4468-8323-4957261063c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:33.686563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6726baf0-67cd-4468-8323-4957261063c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:33.686804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6726baf0-67cd-4468-8323-4957261063c6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:33.687478 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6726baf0-67cd-4468-8323-4957261063c6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:21:33.687858 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6726baf0-67cd-4468-8323-4957261063c6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:33.695795 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6726baf0-67cd-4468-8323-4957261063c6 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:33.774890 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1106/2222] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:21:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:21:33.779208 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b690a6b-65dd-4d1f-ac25-6ba024129564 service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:33.812665 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:33.826432 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b690a6b-65dd-4d1f-ac25-6ba024129564 service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:33.854362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6a5f50-052a-44b0-8f2a-8f1bc2f33526 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:33.855431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6a5f50-052a-44b0-8f2a-8f1bc2f33526 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:33.855720 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6a5f50-052a-44b0-8f2a-8f1bc2f33526 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:33.856001 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6a5f50-052a-44b0-8f2a-8f1bc2f33526 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:33.856781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6a5f50-052a-44b0-8f2a-8f1bc2f33526 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:33.859542 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6a5f50-052a-44b0-8f2a-8f1bc2f33526 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:33.945342 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1117/2223] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:21:33 2022] POST /identity/v3/projects => generated 375 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:21:33.948887 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6726baf0-67cd-4468-8323-4957261063c6 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:33.964983 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6726baf0-67cd-4468-8323-4957261063c6 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:34.098664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:34.113899 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214b2fbc-2f3f-4a3d-b1e6-4da52ed082d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:34.114523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214b2fbc-2f3f-4a3d-b1e6-4da52ed082d6 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:34.114832 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214b2fbc-2f3f-4a3d-b1e6-4da52ed082d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:34.115074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214b2fbc-2f3f-4a3d-b1e6-4da52ed082d6 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:34.115725 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214b2fbc-2f3f-4a3d-b1e6-4da52ed082d6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:34.117931 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214b2fbc-2f3f-4a3d-b1e6-4da52ed082d6 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:34.173977 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1107/2224] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:21:33 2022] POST /identity/v3/users => generated 288 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:21:34.178321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6a5f50-052a-44b0-8f2a-8f1bc2f33526 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:34.193837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6a5f50-052a-44b0-8f2a-8f1bc2f33526 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:34.199469 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf6810e-048f-4f7a-9b02-8dd9091878ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:34.200440 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf6810e-048f-4f7a-9b02-8dd9091878ef admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:34.200877 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf6810e-048f-4f7a-9b02-8dd9091878ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:34.201315 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf6810e-048f-4f7a-9b02-8dd9091878ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:34.202695 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf6810e-048f-4f7a-9b02-8dd9091878ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:34.205052 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf6810e-048f-4f7a-9b02-8dd9091878ef admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:34.220654 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1118/2225] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:34 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:21:34.224436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214b2fbc-2f3f-4a3d-b1e6-4da52ed082d6 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:34.237560 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214b2fbc-2f3f-4a3d-b1e6-4da52ed082d6 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:34.243748 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c8d7d8-1cd1-4cfe-8fc6-004f43db39ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:34.244366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c8d7d8-1cd1-4cfe-8fc6-004f43db39ed admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:34.244649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c8d7d8-1cd1-4cfe-8fc6-004f43db39ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:34.244925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c8d7d8-1cd1-4cfe-8fc6-004f43db39ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:34.245582 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c8d7d8-1cd1-4cfe-8fc6-004f43db39ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:34.251260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c8d7d8-1cd1-4cfe-8fc6-004f43db39ed admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:34.268815 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1108/2226] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:34 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:21:34.270110 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf6810e-048f-4f7a-9b02-8dd9091878ef admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:34.298142 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf6810e-048f-4f7a-9b02-8dd9091878ef admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:34.308475 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8f38c5-f6d7-4b31-a7aa-9cf3f734edb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:34.309262 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8f38c5-f6d7-4b31-a7aa-9cf3f734edb6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:34.309684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8f38c5-f6d7-4b31-a7aa-9cf3f734edb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:34.310066 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8f38c5-f6d7-4b31-a7aa-9cf3f734edb6 admin admin] PATH_INFO: `/v3/projects/988a2e4b6d154a4da1ee1a1f4d696588/users/ad63c13d1c284221a761ff3550b4c6ca/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:34.316682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c8d7d8-1cd1-4cfe-8fc6-004f43db39ed admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:34.319298 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-12c8d7d8-1cd1-4cfe-8fc6-004f43db39ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:34.336709 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c8d7d8-1cd1-4cfe-8fc6-004f43db39ed admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:34.341904 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca3f5dc-faf9-4195-9cec-d08d75511bcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:34.342760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca3f5dc-faf9-4195-9cec-d08d75511bcb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:34.343042 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca3f5dc-faf9-4195-9cec-d08d75511bcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:34.343484 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca3f5dc-faf9-4195-9cec-d08d75511bcb admin admin] PATH_INFO: `/v3/projects/8ebcf45fe4054d38839569d76df9565f` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:34.344697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8f38c5-f6d7-4b31-a7aa-9cf3f734edb6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=988a2e4b6d154a4da1ee1a1f4d696588, user_id=ad63c13d1c284221a761ff3550b4c6ca, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:34.344833 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca3f5dc-faf9-4195-9cec-d08d75511bcb admin admin] RBAC: Authorizing `identity:delete_project(project_id=8ebcf45fe4054d38839569d76df9565f)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:34.347381 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca3f5dc-faf9-4195-9cec-d08d75511bcb admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:34.347521 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8f38c5-f6d7-4b31-a7aa-9cf3f734edb6 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:34.377186 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1119/2227] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:21:34 2022] PUT /identity/v3/projects/988a2e4b6d154a4da1ee1a1f4d696588/users/ad63c13d1c284221a761ff3550b4c6ca/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:34.388264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68d236c-508a-42b8-97f8-537a054b58bd None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:34.388625 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68d236c-508a-42b8-97f8-537a054b58bd None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:34.388986 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68d236c-508a-42b8-97f8-537a054b58bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:34.416338 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-eca3f5dc-faf9-4195-9cec-d08d75511bcb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8ebcf45fe4054d38839569d76df9565f', 'request_id': 'req-eca3f5dc-faf9-4195-9cec-d08d75511bcb'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:34.426585 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-eca3f5dc-faf9-4195-9cec-d08d75511bcb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8ebcf45fe4054d38839569d76df9565f', 'request_id': 'req-eca3f5dc-faf9-4195-9cec-d08d75511bcb'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:34.432733 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-e68d236c-508a-42b8-97f8-537a054b58bd None None] MFA Rules not processed for user `ad63c13d1c284221a761ff3550b4c6ca`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:34.433655 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-eca3f5dc-faf9-4195-9cec-d08d75511bcb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8ebcf45fe4054d38839569d76df9565f', 'request_id': 'req-eca3f5dc-faf9-4195-9cec-d08d75511bcb'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:34.434114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-e68d236c-508a-42b8-97f8-537a054b58bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:34.450530 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-eca3f5dc-faf9-4195-9cec-d08d75511bcb admin admin] The token cache is being invalidate because project 8ebcf45fe4054d38839569d76df9565f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:21:34.450923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-eca3f5dc-faf9-4195-9cec-d08d75511bcb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:34.451785 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-eca3f5dc-faf9-4195-9cec-d08d75511bcb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:34.454401 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1109/2228] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:21:34 2022] DELETE /identity/v3/projects/8ebcf45fe4054d38839569d76df9565f => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:21:34.459974 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dfebd7-e805-48a3-ae22-e0321d3c8e3d None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:34.460452 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dfebd7-e805-48a3-ae22-e0321d3c8e3d None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:34.460729 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dfebd7-e805-48a3-ae22-e0321d3c8e3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:34.485822 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-48dfebd7-e805-48a3-ae22-e0321d3c8e3d None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:34.487068 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-48dfebd7-e805-48a3-ae22-e0321d3c8e3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:34.521967 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1120/2229] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:34 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:34.531144 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68d236c-508a-42b8-97f8-537a054b58bd None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:34.534287 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-e68d236c-508a-42b8-97f8-537a054b58bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:34.575908 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1110/2230] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:21:34.584112 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795f07d1-4754-4e9a-b2c0-b75ba6dba692 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:34.585355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795f07d1-4754-4e9a-b2c0-b75ba6dba692 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:34.586008 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795f07d1-4754-4e9a-b2c0-b75ba6dba692 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:34.607686 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-795f07d1-4754-4e9a-b2c0-b75ba6dba692 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:34.609213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-795f07d1-4754-4e9a-b2c0-b75ba6dba692 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:34.613477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68d236c-508a-42b8-97f8-537a054b58bd None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:34.620837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03558557-ef76-4a60-821d-c24be4494a0e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:34.621045 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1111/2231] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:21:34.621150 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03558557-ef76-4a60-821d-c24be4494a0e service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:34.621150 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03558557-ef76-4a60-821d-c24be4494a0e service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:34.621150 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03558557-ef76-4a60-821d-c24be4494a0e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:34.621894 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795f07d1-4754-4e9a-b2c0-b75ba6dba692 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:34.630768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03558557-ef76-4a60-821d-c24be4494a0e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:34.633196 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03558557-ef76-4a60-821d-c24be4494a0e service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:34.634494 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795f07d1-4754-4e9a-b2c0-b75ba6dba692 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:34.645222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc600e67-6467-4963-99ba-42c0c04c23c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:34.646444 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1121/2232] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:34 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:21:34.646752 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc600e67-6467-4963-99ba-42c0c04c23c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:34.647103 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc600e67-6467-4963-99ba-42c0c04c23c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:34.647441 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc600e67-6467-4963-99ba-42c0c04c23c7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:34.648308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc600e67-6467-4963-99ba-42c0c04c23c7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:21:34.648643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc600e67-6467-4963-99ba-42c0c04c23c7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:34.650325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc600e67-6467-4963-99ba-42c0c04c23c7 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:34.676465 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1112/2233] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:21:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:21:34.677831 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03558557-ef76-4a60-821d-c24be4494a0e service cinder] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:34.692471 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03558557-ef76-4a60-821d-c24be4494a0e service cinder] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:34.697835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d7d654-fb8f-4d28-b170-85eba4b38675 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:34.699552 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d7d654-fb8f-4d28-b170-85eba4b38675 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:34.699552 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d7d654-fb8f-4d28-b170-85eba4b38675 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:34.699887 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d7d654-fb8f-4d28-b170-85eba4b38675 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:34.700619 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d7d654-fb8f-4d28-b170-85eba4b38675 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:34.702739 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d7d654-fb8f-4d28-b170-85eba4b38675 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:34.747968 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1122/2234] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:21:34 2022] POST /identity/v3/projects => generated 383 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:21:34.749726 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc600e67-6467-4963-99ba-42c0c04c23c7 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:34.763144 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc600e67-6467-4963-99ba-42c0c04c23c7 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:34.805839 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:34.845146 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7ceb21-8b32-4f19-8682-d6b3f2e68bcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:34.846017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7ceb21-8b32-4f19-8682-d6b3f2e68bcd admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:34.846394 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7ceb21-8b32-4f19-8682-d6b3f2e68bcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:34.846876 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7ceb21-8b32-4f19-8682-d6b3f2e68bcd admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:34.847646 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7ceb21-8b32-4f19-8682-d6b3f2e68bcd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:34.850081 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7ceb21-8b32-4f19-8682-d6b3f2e68bcd admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:34.879734 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1113/2235] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:21:34 2022] POST /identity/v3/users => generated 292 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:21:34.883597 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d7d654-fb8f-4d28-b170-85eba4b38675 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:34.897147 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d7d654-fb8f-4d28-b170-85eba4b38675 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:34.902468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597845ee-e50e-443a-805d-2149e24a92da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:34.903238 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597845ee-e50e-443a-805d-2149e24a92da admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:34.903721 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597845ee-e50e-443a-805d-2149e24a92da admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:34.904088 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597845ee-e50e-443a-805d-2149e24a92da admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:34.905288 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597845ee-e50e-443a-805d-2149e24a92da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:34.907159 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597845ee-e50e-443a-805d-2149e24a92da admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:34.919890 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1123/2236] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:34 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:21:34.924189 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7ceb21-8b32-4f19-8682-d6b3f2e68bcd admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:34.935230 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7ceb21-8b32-4f19-8682-d6b3f2e68bcd admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:34.942136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b22cbc-69a6-4dc0-a020-29908d1a017e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:34.942740 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b22cbc-69a6-4dc0-a020-29908d1a017e admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:34.943022 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b22cbc-69a6-4dc0-a020-29908d1a017e admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:34.943305 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b22cbc-69a6-4dc0-a020-29908d1a017e admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:34.943984 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b22cbc-69a6-4dc0-a020-29908d1a017e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:34.945575 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b22cbc-69a6-4dc0-a020-29908d1a017e admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:34.961827 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1114/2237] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:34 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:21:34.965687 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597845ee-e50e-443a-805d-2149e24a92da admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:34.974375 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:34.979310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597845ee-e50e-443a-805d-2149e24a92da admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:34.987684 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee212c0-720c-4f79-b383-797c67363ce5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:34.988496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee212c0-720c-4f79-b383-797c67363ce5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:34.988841 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee212c0-720c-4f79-b383-797c67363ce5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:34.989274 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee212c0-720c-4f79-b383-797c67363ce5 admin admin] PATH_INFO: `/v3/projects/8ef89153c2d24ff99040819c56e54e8c/users/47fb56aeb85f48e883d3ccb7f0d4469c/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:35.027556 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee212c0-720c-4f79-b383-797c67363ce5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ef89153c2d24ff99040819c56e54e8c, user_id=47fb56aeb85f48e883d3ccb7f0d4469c, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:35.038665 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee212c0-720c-4f79-b383-797c67363ce5 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:35.066918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1124/2238] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:21:34 2022] PUT /identity/v3/projects/8ef89153c2d24ff99040819c56e54e8c/users/47fb56aeb85f48e883d3ccb7f0d4469c/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:35.076880 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce95468b-1d60-4e8a-9e0d-54ee00cc6dc1 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:35.077213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce95468b-1d60-4e8a-9e0d-54ee00cc6dc1 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:35.077499 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce95468b-1d60-4e8a-9e0d-54ee00cc6dc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:35.132878 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-ce95468b-1d60-4e8a-9e0d-54ee00cc6dc1 None None] MFA Rules not processed for user `47fb56aeb85f48e883d3ccb7f0d4469c`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:35.134296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ce95468b-1d60-4e8a-9e0d-54ee00cc6dc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:35.224938 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1115/2239] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:35 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:35.226566 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee212c0-720c-4f79-b383-797c67363ce5 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:35.305211 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee212c0-720c-4f79-b383-797c67363ce5 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:35.310318 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef503b87-ed01-4d5d-9923-7507c9248151 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:35.311180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef503b87-ed01-4d5d-9923-7507c9248151 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:35.311680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef503b87-ed01-4d5d-9923-7507c9248151 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:35.312143 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef503b87-ed01-4d5d-9923-7507c9248151 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:35.312972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef503b87-ed01-4d5d-9923-7507c9248151 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:35.315104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef503b87-ed01-4d5d-9923-7507c9248151 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:35.364175 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1125/2240] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:21:35 2022] POST /identity/v3/projects => generated 383 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:21:35.368032 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce95468b-1d60-4e8a-9e0d-54ee00cc6dc1 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:35.384980 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce95468b-1d60-4e8a-9e0d-54ee00cc6dc1 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:35.483693 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badc745e-d187-4b19-b0fe-1e6d3accc7fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:35.484480 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badc745e-d187-4b19-b0fe-1e6d3accc7fc admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:35.484859 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badc745e-d187-4b19-b0fe-1e6d3accc7fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:35.485150 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badc745e-d187-4b19-b0fe-1e6d3accc7fc admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:35.485783 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badc745e-d187-4b19-b0fe-1e6d3accc7fc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:35.487700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badc745e-d187-4b19-b0fe-1e6d3accc7fc admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:35.518715 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1116/2241] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:21:35 2022] POST /identity/v3/users => generated 292 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:21:35.523004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef503b87-ed01-4d5d-9923-7507c9248151 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:35.537697 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef503b87-ed01-4d5d-9923-7507c9248151 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:35.544055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5895cd4c-1fc1-4b71-8b6b-d91e8b4e1994 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:35.544651 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5895cd4c-1fc1-4b71-8b6b-d91e8b4e1994 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:35.544922 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5895cd4c-1fc1-4b71-8b6b-d91e8b4e1994 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:35.545185 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5895cd4c-1fc1-4b71-8b6b-d91e8b4e1994 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:35.545912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5895cd4c-1fc1-4b71-8b6b-d91e8b4e1994 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:35.547491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5895cd4c-1fc1-4b71-8b6b-d91e8b4e1994 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:35.565409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badc745e-d187-4b19-b0fe-1e6d3accc7fc admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:35.565859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1126/2242] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:35 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:21:35.576817 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badc745e-d187-4b19-b0fe-1e6d3accc7fc admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:35.585788 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8d66cb-4337-45b6-893a-5258d7e96354 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:35.586412 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8d66cb-4337-45b6-893a-5258d7e96354 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:35.586752 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8d66cb-4337-45b6-893a-5258d7e96354 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:35.587035 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8d66cb-4337-45b6-893a-5258d7e96354 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:35.587749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8d66cb-4337-45b6-893a-5258d7e96354 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:35.589715 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8d66cb-4337-45b6-893a-5258d7e96354 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:35.603099 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1117/2243] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:35 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:21:35.607567 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5895cd4c-1fc1-4b71-8b6b-d91e8b4e1994 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:35.618461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5895cd4c-1fc1-4b71-8b6b-d91e8b4e1994 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:35.626448 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79723aee-4f37-4aa8-887c-24df4dd3fdc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:35.627997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79723aee-4f37-4aa8-887c-24df4dd3fdc3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:35.628539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79723aee-4f37-4aa8-887c-24df4dd3fdc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:35.629530 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79723aee-4f37-4aa8-887c-24df4dd3fdc3 admin admin] PATH_INFO: `/v3/projects/7de4e6ebc0554132a6ad5ee20c9c44d6/users/287ad17db4534a65b3b0850a2ba41117/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:35.654184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79723aee-4f37-4aa8-887c-24df4dd3fdc3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7de4e6ebc0554132a6ad5ee20c9c44d6, user_id=287ad17db4534a65b3b0850a2ba41117, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:35.658974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79723aee-4f37-4aa8-887c-24df4dd3fdc3 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:35.680289 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1127/2244] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:21:35 2022] PUT /identity/v3/projects/7de4e6ebc0554132a6ad5ee20c9c44d6/users/287ad17db4534a65b3b0850a2ba41117/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:35.691573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13673b39-d8a2-4bef-9462-e46f82b7ff65 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:35.691869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13673b39-d8a2-4bef-9462-e46f82b7ff65 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:35.692150 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13673b39-d8a2-4bef-9462-e46f82b7ff65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:35.725867 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-13673b39-d8a2-4bef-9462-e46f82b7ff65 None None] MFA Rules not processed for user `287ad17db4534a65b3b0850a2ba41117`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:35.727089 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-13673b39-d8a2-4bef-9462-e46f82b7ff65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:35.737205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:35.801846 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1118/2245] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:35 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:35.804565 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79723aee-4f37-4aa8-887c-24df4dd3fdc3 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:35.874574 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79723aee-4f37-4aa8-887c-24df4dd3fdc3 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:35.881884 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef6234c-5ed3-4116-9360-b3eccb7e25be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:35.882494 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef6234c-5ed3-4116-9360-b3eccb7e25be admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:35.882769 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef6234c-5ed3-4116-9360-b3eccb7e25be admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:35.883039 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef6234c-5ed3-4116-9360-b3eccb7e25be admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:35.883667 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef6234c-5ed3-4116-9360-b3eccb7e25be admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:35.885681 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef6234c-5ed3-4116-9360-b3eccb7e25be admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:35.917864 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1128/2246] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:21:35 2022] POST /identity/v3/projects => generated 383 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:21:35.921552 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13673b39-d8a2-4bef-9462-e46f82b7ff65 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:35.938336 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13673b39-d8a2-4bef-9462-e46f82b7ff65 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:36.008226 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28476d30-960c-4afd-82ad-b59254fd5354 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:36.009113 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28476d30-960c-4afd-82ad-b59254fd5354 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:36.009487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28476d30-960c-4afd-82ad-b59254fd5354 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:36.009930 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28476d30-960c-4afd-82ad-b59254fd5354 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:36.010666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28476d30-960c-4afd-82ad-b59254fd5354 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:36.012681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28476d30-960c-4afd-82ad-b59254fd5354 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:36.042246 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1119/2247] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:21:35 2022] POST /identity/v3/users => generated 298 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:21:36.042466 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef6234c-5ed3-4116-9360-b3eccb7e25be admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:36.057553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef6234c-5ed3-4116-9360-b3eccb7e25be admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:36.065810 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fba2b2-a24d-485f-92d7-387c80159116 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:36.070064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fba2b2-a24d-485f-92d7-387c80159116 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:36.070441 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fba2b2-a24d-485f-92d7-387c80159116 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:36.070814 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fba2b2-a24d-485f-92d7-387c80159116 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:36.071643 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75fba2b2-a24d-485f-92d7-387c80159116 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:36.073645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75fba2b2-a24d-485f-92d7-387c80159116 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:36.090757 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1129/2248] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:36 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:21:36.097955 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28476d30-960c-4afd-82ad-b59254fd5354 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:36.115975 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28476d30-960c-4afd-82ad-b59254fd5354 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:36.124470 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7d6dd6-f4d3-487e-8f85-eee28227b71e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:36.125718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7d6dd6-f4d3-487e-8f85-eee28227b71e admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:36.126038 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7d6dd6-f4d3-487e-8f85-eee28227b71e admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:36.126675 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7d6dd6-f4d3-487e-8f85-eee28227b71e admin admin] PATH_INFO: `/v3/projects/cb383707cc014e58ad6685429e2f3388/users/e19b722cf8d049fa83d8f18ae67ff664/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:36.165767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7d6dd6-f4d3-487e-8f85-eee28227b71e admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb383707cc014e58ad6685429e2f3388, user_id=e19b722cf8d049fa83d8f18ae67ff664, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:36.165767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7d6dd6-f4d3-487e-8f85-eee28227b71e admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:36.188179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1120/2249] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:21:36 2022] PUT /identity/v3/projects/cb383707cc014e58ad6685429e2f3388/users/e19b722cf8d049fa83d8f18ae67ff664/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:21:36.189378 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fba2b2-a24d-485f-92d7-387c80159116 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:36.196005 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:36.202039 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fba2b2-a24d-485f-92d7-387c80159116 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:36.266545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5b38c1-075f-42e9-b2f6-bb8ca66f8864 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:36.267154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5b38c1-075f-42e9-b2f6-bb8ca66f8864 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:36.267432 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5b38c1-075f-42e9-b2f6-bb8ca66f8864 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:36.267699 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5b38c1-075f-42e9-b2f6-bb8ca66f8864 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:36.268353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5b38c1-075f-42e9-b2f6-bb8ca66f8864 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:36.269963 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5b38c1-075f-42e9-b2f6-bb8ca66f8864 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:36.281790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1130/2250] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:36 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:21:36.285453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7d6dd6-f4d3-487e-8f85-eee28227b71e admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:36.297030 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7d6dd6-f4d3-487e-8f85-eee28227b71e admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:36.303339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03138142-56f5-4519-b8ea-a53a6063f799 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:36.304067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03138142-56f5-4519-b8ea-a53a6063f799 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:36.304450 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03138142-56f5-4519-b8ea-a53a6063f799 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:36.304822 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03138142-56f5-4519-b8ea-a53a6063f799 admin admin] PATH_INFO: `/v3/projects/cb383707cc014e58ad6685429e2f3388/users/e19b722cf8d049fa83d8f18ae67ff664/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:36.307037 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03138142-56f5-4519-b8ea-a53a6063f799 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb383707cc014e58ad6685429e2f3388, user_id=e19b722cf8d049fa83d8f18ae67ff664, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:36.309972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03138142-56f5-4519-b8ea-a53a6063f799 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:36.345101 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1121/2251] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:21:36 2022] PUT /identity/v3/projects/cb383707cc014e58ad6685429e2f3388/users/e19b722cf8d049fa83d8f18ae67ff664/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:36.348124 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9f20f1-2478-40dd-b1b2-806bd680c125 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:36.348507 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9f20f1-2478-40dd-b1b2-806bd680c125 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:36.349044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9f20f1-2478-40dd-b1b2-806bd680c125 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:36.407870 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-dd9f20f1-2478-40dd-b1b2-806bd680c125 None None] MFA Rules not processed for user `e19b722cf8d049fa83d8f18ae67ff664`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:36.409104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-dd9f20f1-2478-40dd-b1b2-806bd680c125 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:36.554948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1131/2252] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:36 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:36.567673 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03138142-56f5-4519-b8ea-a53a6063f799 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:36.709819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03138142-56f5-4519-b8ea-a53a6063f799 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:36.715360 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245e84a1-826a-42f4-b7ce-bc9800bee1ca service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:36.717012 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245e84a1-826a-42f4-b7ce-bc9800bee1ca service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:36.717299 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245e84a1-826a-42f4-b7ce-bc9800bee1ca service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:36.718798 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245e84a1-826a-42f4-b7ce-bc9800bee1ca service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:36.763171 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:36.825143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245e84a1-826a-42f4-b7ce-bc9800bee1ca service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:36.827360 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245e84a1-826a-42f4-b7ce-bc9800bee1ca service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:36.840370 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1122/2253] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:36 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 273 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:21:39.448161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675b30bb-fc1e-41b9-85de-8bafc6257e9f None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:39.448161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675b30bb-fc1e-41b9-85de-8bafc6257e9f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:39.448968 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675b30bb-fc1e-41b9-85de-8bafc6257e9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:39.479129 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-675b30bb-fc1e-41b9-85de-8bafc6257e9f None None] MFA Rules not processed for user `ad63c13d1c284221a761ff3550b4c6ca`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:39.480453 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-675b30bb-fc1e-41b9-85de-8bafc6257e9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:39.573609 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1132/2254] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:39 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:40.775160 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245e84a1-826a-42f4-b7ce-bc9800bee1ca service cinder] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:40.786904 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245e84a1-826a-42f4-b7ce-bc9800bee1ca service cinder] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:40.791606 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667615ef-9589-4479-8357-bdd4dfdeabe8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:40.792330 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667615ef-9589-4479-8357-bdd4dfdeabe8 service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:40.792704 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667615ef-9589-4479-8357-bdd4dfdeabe8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:40.792994 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667615ef-9589-4479-8357-bdd4dfdeabe8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:40.920854 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675b30bb-fc1e-41b9-85de-8bafc6257e9f None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:40.922583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-675b30bb-fc1e-41b9-85de-8bafc6257e9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:40.956876 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667615ef-9589-4479-8357-bdd4dfdeabe8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:40.958667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667615ef-9589-4479-8357-bdd4dfdeabe8 service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:40.980580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1123/2255] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:40 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 205 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:21:40.989996 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:41.038620 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675b30bb-fc1e-41b9-85de-8bafc6257e9f None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:41.044221 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11875a5c-3fe9-4e22-9025-91398c6a7862 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:41.045185 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11875a5c-3fe9-4e22-9025-91398c6a7862 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:41.045640 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11875a5c-3fe9-4e22-9025-91398c6a7862 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:41.045984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11875a5c-3fe9-4e22-9025-91398c6a7862 admin admin] PATH_INFO: `/v3/users/f49663f1fd5a47b5bce5cee57ce2f4b0` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:41.047496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11875a5c-3fe9-4e22-9025-91398c6a7862 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f49663f1fd5a47b5bce5cee57ce2f4b0)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:41.050996 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11875a5c-3fe9-4e22-9025-91398c6a7862 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:41.129718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-11875a5c-3fe9-4e22-9025-91398c6a7862 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f49663f1fd5a47b5bce5cee57ce2f4b0'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:41.142255 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-11875a5c-3fe9-4e22-9025-91398c6a7862 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f49663f1fd5a47b5bce5cee57ce2f4b0'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:41.142957 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-11875a5c-3fe9-4e22-9025-91398c6a7862 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f49663f1fd5a47b5bce5cee57ce2f4b0'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:41.155851 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-11875a5c-3fe9-4e22-9025-91398c6a7862 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f49663f1fd5a47b5bce5cee57ce2f4b0'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:41.196232 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-11875a5c-3fe9-4e22-9025-91398c6a7862 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f49663f1fd5a47b5bce5cee57ce2f4b0'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:41.217467 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1133/2256] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:21:40 2022] DELETE /identity/v3/users/f49663f1fd5a47b5bce5cee57ce2f4b0 => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:41.218571 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfb4808-9a22-41ef-885d-d01dd8461308 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:41.218872 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfb4808-9a22-41ef-885d-d01dd8461308 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:41.219814 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfb4808-9a22-41ef-885d-d01dd8461308 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:41.247998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-ddfb4808-9a22-41ef-885d-d01dd8461308 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:41.249644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ddfb4808-9a22-41ef-885d-d01dd8461308 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:41.351648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1124/2257] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:21:41.359460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11875a5c-3fe9-4e22-9025-91398c6a7862 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:41.361534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-11875a5c-3fe9-4e22-9025-91398c6a7862 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:41.459362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11875a5c-3fe9-4e22-9025-91398c6a7862 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:41.492624 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb6e7ab-988f-4f9b-a862-189084d02afb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:41.493727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb6e7ab-988f-4f9b-a862-189084d02afb service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:41.494323 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb6e7ab-988f-4f9b-a862-189084d02afb service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:41.494914 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb6e7ab-988f-4f9b-a862-189084d02afb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:41.508022 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb6e7ab-988f-4f9b-a862-189084d02afb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:41.510548 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb6e7ab-988f-4f9b-a862-189084d02afb service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:41.559034 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1134/2258] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:21:41.809389 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0266350e-9bba-45f2-8d7c-2f103e27481a None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:41.809910 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0266350e-9bba-45f2-8d7c-2f103e27481a None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:41.809999 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb6e7ab-988f-4f9b-a862-189084d02afb service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:41.810099 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0266350e-9bba-45f2-8d7c-2f103e27481a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:41.811933 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-5cb6e7ab-988f-4f9b-a862-189084d02afb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:41.833457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-0266350e-9bba-45f2-8d7c-2f103e27481a None None] MFA Rules not processed for user `42d91d56296e4fdd917495df41401921`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:41.835276 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0266350e-9bba-45f2-8d7c-2f103e27481a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:41.889196 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb6e7ab-988f-4f9b-a862-189084d02afb service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:41.893995 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33cb7de-8a6c-40af-ae49-664c2dc589f4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:41.894728 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33cb7de-8a6c-40af-ae49-664c2dc589f4 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:41.895404 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33cb7de-8a6c-40af-ae49-664c2dc589f4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:41.895993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33cb7de-8a6c-40af-ae49-664c2dc589f4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:41.898002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-b33cb7de-8a6c-40af-ae49-664c2dc589f4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:41.905711 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:41.931118 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1125/2259] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:41 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:41.995022 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33cb7de-8a6c-40af-ae49-664c2dc589f4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:41.996566 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33cb7de-8a6c-40af-ae49-664c2dc589f4 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:42.009070 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1135/2260] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:41 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:21:42.039728 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0266350e-9bba-45f2-8d7c-2f103e27481a None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:42.041393 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0266350e-9bba-45f2-8d7c-2f103e27481a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:42.050477 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:42.059957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0266350e-9bba-45f2-8d7c-2f103e27481a None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:42.076040 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577bebf6-5e64-4265-bbd2-edf9836646fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:42.076764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577bebf6-5e64-4265-bbd2-edf9836646fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:42.078321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577bebf6-5e64-4265-bbd2-edf9836646fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:42.078616 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577bebf6-5e64-4265-bbd2-edf9836646fa admin admin] PATH_INFO: `/v3/projects/118a051914be4932887e976dab9efdf5` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:42.082079 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577bebf6-5e64-4265-bbd2-edf9836646fa admin admin] RBAC: Authorizing `identity:delete_project(project_id=118a051914be4932887e976dab9efdf5)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:42.086156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577bebf6-5e64-4265-bbd2-edf9836646fa admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:42.205813 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-577bebf6-5e64-4265-bbd2-edf9836646fa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '118a051914be4932887e976dab9efdf5', 'request_id': 'req-577bebf6-5e64-4265-bbd2-edf9836646fa'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:42.205813 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-577bebf6-5e64-4265-bbd2-edf9836646fa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '118a051914be4932887e976dab9efdf5', 'request_id': 'req-577bebf6-5e64-4265-bbd2-edf9836646fa'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:42.220960 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-577bebf6-5e64-4265-bbd2-edf9836646fa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '118a051914be4932887e976dab9efdf5', 'request_id': 'req-577bebf6-5e64-4265-bbd2-edf9836646fa'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:42.249645 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-577bebf6-5e64-4265-bbd2-edf9836646fa admin admin] The token cache is being invalidate because project 118a051914be4932887e976dab9efdf5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:21:42.249942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-577bebf6-5e64-4265-bbd2-edf9836646fa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:42.250622 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-577bebf6-5e64-4265-bbd2-edf9836646fa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:42.257603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1126/2261] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:21:42 2022] DELETE /identity/v3/projects/118a051914be4932887e976dab9efdf5 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:42.259639 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbdd544-5445-434c-b9a8-9a535390e738 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:42.259935 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbdd544-5445-434c-b9a8-9a535390e738 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:42.260202 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbdd544-5445-434c-b9a8-9a535390e738 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:42.300780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-2cbdd544-5445-434c-b9a8-9a535390e738 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:42.303545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-2cbdd544-5445-434c-b9a8-9a535390e738 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:42.451255 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1136/2262] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:21:42.459816 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9c2268-6282-4881-be6e-da6218c20dd0 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:42.460108 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9c2268-6282-4881-be6e-da6218c20dd0 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:42.460389 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9c2268-6282-4881-be6e-da6218c20dd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:42.490872 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-ad9c2268-6282-4881-be6e-da6218c20dd0 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:42.493155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ad9c2268-6282-4881-be6e-da6218c20dd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:42.503923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1127/2263] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:42.508673 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbdd544-5445-434c-b9a8-9a535390e738 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:42.520146 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbdd544-5445-434c-b9a8-9a535390e738 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:42.554981 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15c95e3-cc98-48b7-b174-b259338b7b48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:42.556133 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15c95e3-cc98-48b7-b174-b259338b7b48 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:42.556422 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15c95e3-cc98-48b7-b174-b259338b7b48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:42.556699 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15c95e3-cc98-48b7-b174-b259338b7b48 admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:42.557433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15c95e3-cc98-48b7-b174-b259338b7b48 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:21:42.559569 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15c95e3-cc98-48b7-b174-b259338b7b48 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:42.562193 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15c95e3-cc98-48b7-b174-b259338b7b48 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:42.595108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1137/2264] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:21:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:21:42.599723 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9c2268-6282-4881-be6e-da6218c20dd0 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:42.627222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9c2268-6282-4881-be6e-da6218c20dd0 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:42.641632 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e9fc96-b101-4739-aecc-79b9ad77ae9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:42.641632 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e9fc96-b101-4739-aecc-79b9ad77ae9e admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:42.641632 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e9fc96-b101-4739-aecc-79b9ad77ae9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:42.641632 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e9fc96-b101-4739-aecc-79b9ad77ae9e admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:42.642069 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e9fc96-b101-4739-aecc-79b9ad77ae9e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:42.651668 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e9fc96-b101-4739-aecc-79b9ad77ae9e admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:42.721032 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1128/2265] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:21:42 2022] POST /identity/v3/projects => generated 383 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:21:42.725102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15c95e3-cc98-48b7-b174-b259338b7b48 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:42.738708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15c95e3-cc98-48b7-b174-b259338b7b48 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:42.817171 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bad50e-29ff-4160-bf45-c1a026d1d6c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:42.817847 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bad50e-29ff-4160-bf45-c1a026d1d6c9 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:42.818250 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bad50e-29ff-4160-bf45-c1a026d1d6c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:42.818553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bad50e-29ff-4160-bf45-c1a026d1d6c9 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:42.819192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89bad50e-29ff-4160-bf45-c1a026d1d6c9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:42.821270 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89bad50e-29ff-4160-bf45-c1a026d1d6c9 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:42.859903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1138/2266] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:21:42 2022] POST /identity/v3/users => generated 292 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:21:42.862011 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e9fc96-b101-4739-aecc-79b9ad77ae9e admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:42.875008 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e9fc96-b101-4739-aecc-79b9ad77ae9e admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:42.879941 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f79a16f-9ed8-4041-96d9-f5110c2a7a6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:42.880847 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f79a16f-9ed8-4041-96d9-f5110c2a7a6d admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:42.881214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f79a16f-9ed8-4041-96d9-f5110c2a7a6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:42.881551 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f79a16f-9ed8-4041-96d9-f5110c2a7a6d admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:42.882317 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f79a16f-9ed8-4041-96d9-f5110c2a7a6d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:42.884136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f79a16f-9ed8-4041-96d9-f5110c2a7a6d admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:42.897796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1129/2267] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:42 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:21:42.900443 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bad50e-29ff-4160-bf45-c1a026d1d6c9 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:42.911983 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bad50e-29ff-4160-bf45-c1a026d1d6c9 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:42.919390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8201199-8ce5-4207-b995-ec1d0a883ea4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:42.920104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8201199-8ce5-4207-b995-ec1d0a883ea4 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:42.920504 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8201199-8ce5-4207-b995-ec1d0a883ea4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:42.921408 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8201199-8ce5-4207-b995-ec1d0a883ea4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:42.922336 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8201199-8ce5-4207-b995-ec1d0a883ea4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:42.924474 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8201199-8ce5-4207-b995-ec1d0a883ea4 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:42.947046 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1139/2268] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:42 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:21:42.951504 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f79a16f-9ed8-4041-96d9-f5110c2a7a6d admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:42.962716 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f79a16f-9ed8-4041-96d9-f5110c2a7a6d admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:42.968174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abb8c4c-4176-4b47-b9b8-04af4070c16b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:42.969004 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abb8c4c-4176-4b47-b9b8-04af4070c16b admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:42.969354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abb8c4c-4176-4b47-b9b8-04af4070c16b admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:42.969696 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abb8c4c-4176-4b47-b9b8-04af4070c16b admin admin] PATH_INFO: `/v3/projects/cd836c63e9f14336a59ea9d51b05510b/users/b636bbcb12d74f229c80748885cfbdb7/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:42.990936 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3abb8c4c-4176-4b47-b9b8-04af4070c16b admin admin] RBAC: Authorizing `identity:create_grant(project_id=cd836c63e9f14336a59ea9d51b05510b, user_id=b636bbcb12d74f229c80748885cfbdb7, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:42.993690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3abb8c4c-4176-4b47-b9b8-04af4070c16b admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:43.009838 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1130/2269] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:21:42 2022] PUT /identity/v3/projects/cd836c63e9f14336a59ea9d51b05510b/users/b636bbcb12d74f229c80748885cfbdb7/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:43.020174 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de9d077-f1ee-469a-941b-d39d7313a6f2 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:43.020624 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de9d077-f1ee-469a-941b-d39d7313a6f2 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:43.020870 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de9d077-f1ee-469a-941b-d39d7313a6f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:43.045227 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:43.057977 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-3de9d077-f1ee-469a-941b-d39d7313a6f2 None None] MFA Rules not processed for user `b636bbcb12d74f229c80748885cfbdb7`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:43.059208 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-3de9d077-f1ee-469a-941b-d39d7313a6f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:43.144857 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1140/2270] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:43 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:43.153297 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abb8c4c-4176-4b47-b9b8-04af4070c16b admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:43.155301 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-3abb8c4c-4176-4b47-b9b8-04af4070c16b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:43.237065 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abb8c4c-4176-4b47-b9b8-04af4070c16b admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:43.243306 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c801530-2b14-474f-85bc-eb8a799125ee service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:43.244246 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c801530-2b14-474f-85bc-eb8a799125ee service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:43.244779 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c801530-2b14-474f-85bc-eb8a799125ee service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:43.245287 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c801530-2b14-474f-85bc-eb8a799125ee service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:43.256684 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c801530-2b14-474f-85bc-eb8a799125ee service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:43.258326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c801530-2b14-474f-85bc-eb8a799125ee service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:43.282174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1131/2271] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:43 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:21:47.177725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b918c73c-af5b-4c76-b51d-2a87954ba0f6 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:47.179092 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b918c73c-af5b-4c76-b51d-2a87954ba0f6 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:47.180238 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b918c73c-af5b-4c76-b51d-2a87954ba0f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:47.210845 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-b918c73c-af5b-4c76-b51d-2a87954ba0f6 None None] MFA Rules not processed for user `5a2fcc8c9ac44dceadcc30d163bbfa0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:47.212235 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-b918c73c-af5b-4c76-b51d-2a87954ba0f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:47.314897 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1141/2272] 10.222.0.26 () {58 vars in 1009 bytes} [Tue Sep 27 04:21:47 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:47.329202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c801530-2b14-474f-85bc-eb8a799125ee service cinder] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:47.339106 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-7c801530-2b14-474f-85bc-eb8a799125ee service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:47.379579 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:47.456194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c801530-2b14-474f-85bc-eb8a799125ee service cinder] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:47.463169 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-827d208b-bcbf-4432-8ba2-22737ee927e6 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '181dd5dc720c4a97860852062b7b19c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:47.464233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827d208b-bcbf-4432-8ba2-22737ee927e6 service swift] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:47.464682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827d208b-bcbf-4432-8ba2-22737ee927e6 service swift] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:47.465181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827d208b-bcbf-4432-8ba2-22737ee927e6 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:47.477062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-827d208b-bcbf-4432-8ba2-22737ee927e6 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:47.479262 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-827d208b-bcbf-4432-8ba2-22737ee927e6 service swift] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:47.495231 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1132/2273] 10.222.0.26 () {60 vars in 1384 bytes} [Tue Sep 27 04:21:47 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 04:21:48.238885 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b918c73c-af5b-4c76-b51d-2a87954ba0f6 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:48.319673 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b918c73c-af5b-4c76-b51d-2a87954ba0f6 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:48.328672 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240d812e-140a-4b39-a19b-140fbabebbbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:48.329614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240d812e-140a-4b39-a19b-140fbabebbbe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:48.330083 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240d812e-140a-4b39-a19b-140fbabebbbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:48.330500 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240d812e-140a-4b39-a19b-140fbabebbbe admin admin] PATH_INFO: `/v3/users/b636bbcb12d74f229c80748885cfbdb7` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:48.331795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240d812e-140a-4b39-a19b-140fbabebbbe admin admin] RBAC: Authorizing `identity:delete_user(user_id=b636bbcb12d74f229c80748885cfbdb7)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:48.333960 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240d812e-140a-4b39-a19b-140fbabebbbe admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:48.413425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:48.415953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-240d812e-140a-4b39-a19b-140fbabebbbe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b636bbcb12d74f229c80748885cfbdb7'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:48.430557 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-240d812e-140a-4b39-a19b-140fbabebbbe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b636bbcb12d74f229c80748885cfbdb7'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:48.431612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-240d812e-140a-4b39-a19b-140fbabebbbe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b636bbcb12d74f229c80748885cfbdb7'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:48.455511 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-240d812e-140a-4b39-a19b-140fbabebbbe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b636bbcb12d74f229c80748885cfbdb7'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:48.473558 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-240d812e-140a-4b39-a19b-140fbabebbbe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b636bbcb12d74f229c80748885cfbdb7'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:48.473938 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-827d208b-bcbf-4432-8ba2-22737ee927e6 service swift] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:48.475429 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-827d208b-bcbf-4432-8ba2-22737ee927e6 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:48.482629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1142/2274] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:21:48 2022] DELETE /identity/v3/users/b636bbcb12d74f229c80748885cfbdb7 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:48.487496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a88094e-fdeb-40a7-9aca-12eb2363625f None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:48.487859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a88094e-fdeb-40a7-9aca-12eb2363625f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:48.489311 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a88094e-fdeb-40a7-9aca-12eb2363625f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:48.495051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-827d208b-bcbf-4432-8ba2-22737ee927e6 service swift] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:48.513295 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-1a88094e-fdeb-40a7-9aca-12eb2363625f None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:48.514531 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-1a88094e-fdeb-40a7-9aca-12eb2363625f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:48.577818 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c7d3df-2435-4ee9-b036-3c054c24e901 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:48.578476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c7d3df-2435-4ee9-b036-3c054c24e901 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:48.578815 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c7d3df-2435-4ee9-b036-3c054c24e901 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:48.579094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c7d3df-2435-4ee9-b036-3c054c24e901 admin admin] PATH_INFO: `/v3/users/47fb56aeb85f48e883d3ccb7f0d4469c` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:48.582427 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c7d3df-2435-4ee9-b036-3c054c24e901 admin admin] RBAC: Authorizing `identity:delete_user(user_id=47fb56aeb85f48e883d3ccb7f0d4469c)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:48.584981 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c7d3df-2435-4ee9-b036-3c054c24e901 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:48.602102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1143/2275] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:48.615705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a88094e-fdeb-40a7-9aca-12eb2363625f None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:48.618428 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-1a88094e-fdeb-40a7-9aca-12eb2363625f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:48.653275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-38c7d3df-2435-4ee9-b036-3c054c24e901 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '47fb56aeb85f48e883d3ccb7f0d4469c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:48.663951 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-38c7d3df-2435-4ee9-b036-3c054c24e901 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '47fb56aeb85f48e883d3ccb7f0d4469c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:48.665012 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-38c7d3df-2435-4ee9-b036-3c054c24e901 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '47fb56aeb85f48e883d3ccb7f0d4469c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:48.677117 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-38c7d3df-2435-4ee9-b036-3c054c24e901 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '47fb56aeb85f48e883d3ccb7f0d4469c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:48.699912 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-38c7d3df-2435-4ee9-b036-3c054c24e901 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '47fb56aeb85f48e883d3ccb7f0d4469c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:48.713175 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1133/2276] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:21:48 2022] DELETE /identity/v3/users/47fb56aeb85f48e883d3ccb7f0d4469c => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:48.714620 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd011d64-5e61-49c5-8193-3ec854a0394e None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:48.714982 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd011d64-5e61-49c5-8193-3ec854a0394e None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:48.715266 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd011d64-5e61-49c5-8193-3ec854a0394e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:48.715408 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a88094e-fdeb-40a7-9aca-12eb2363625f None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:48.739334 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-bd011d64-5e61-49c5-8193-3ec854a0394e None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:48.740626 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-bd011d64-5e61-49c5-8193-3ec854a0394e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:48.783861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4f70e6-201b-47ce-ad9d-0d934e8133b7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:48.784107 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4f70e6-201b-47ce-ad9d-0d934e8133b7 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:48.784365 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4f70e6-201b-47ce-ad9d-0d934e8133b7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:48.784747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4f70e6-201b-47ce-ad9d-0d934e8133b7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:48.787500 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6a4f70e6-201b-47ce-ad9d-0d934e8133b7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:48.816290 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:48.834539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:48.879099 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1134/2277] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:48.889975 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd011d64-5e61-49c5-8193-3ec854a0394e None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:48.892791 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-bd011d64-5e61-49c5-8193-3ec854a0394e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:48.930010 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd011d64-5e61-49c5-8193-3ec854a0394e None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:48.941421 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a7eb82-2c47-4874-bb38-2ce65345f4d9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:48.943224 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a7eb82-2c47-4874-bb38-2ce65345f4d9 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:48.943747 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a7eb82-2c47-4874-bb38-2ce65345f4d9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:48.944601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4f70e6-201b-47ce-ad9d-0d934e8133b7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:48.946019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a7eb82-2c47-4874-bb38-2ce65345f4d9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:48.947023 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4f70e6-201b-47ce-ad9d-0d934e8133b7 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:48.970273 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a7eb82-2c47-4874-bb38-2ce65345f4d9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:48.974191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a7eb82-2c47-4874-bb38-2ce65345f4d9 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:48.974811 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1144/2278] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:21:48.997893 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1135/2279] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:21:49.854547 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4f70e6-201b-47ce-ad9d-0d934e8133b7 service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:49.856037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6a4f70e6-201b-47ce-ad9d-0d934e8133b7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:49.873295 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4f70e6-201b-47ce-ad9d-0d934e8133b7 service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:49.885797 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c60309-c2bf-43d6-8b26-392e762a64c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:49.887562 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c60309-c2bf-43d6-8b26-392e762a64c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:49.888040 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c60309-c2bf-43d6-8b26-392e762a64c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:49.889782 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c60309-c2bf-43d6-8b26-392e762a64c5 admin admin] PATH_INFO: `/v3/projects/8ef89153c2d24ff99040819c56e54e8c` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:49.902119 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78c60309-c2bf-43d6-8b26-392e762a64c5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8ef89153c2d24ff99040819c56e54e8c)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:49.904301 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78c60309-c2bf-43d6-8b26-392e762a64c5 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:50.008790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-78c60309-c2bf-43d6-8b26-392e762a64c5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8ef89153c2d24ff99040819c56e54e8c', 'request_id': 'req-78c60309-c2bf-43d6-8b26-392e762a64c5'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:50.017030 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-78c60309-c2bf-43d6-8b26-392e762a64c5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8ef89153c2d24ff99040819c56e54e8c', 'request_id': 'req-78c60309-c2bf-43d6-8b26-392e762a64c5'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:50.025643 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-78c60309-c2bf-43d6-8b26-392e762a64c5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8ef89153c2d24ff99040819c56e54e8c', 'request_id': 'req-78c60309-c2bf-43d6-8b26-392e762a64c5'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:50.050659 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-78c60309-c2bf-43d6-8b26-392e762a64c5 admin admin] The token cache is being invalidate because project 8ef89153c2d24ff99040819c56e54e8c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:21:50.051303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-78c60309-c2bf-43d6-8b26-392e762a64c5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:50.052459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-78c60309-c2bf-43d6-8b26-392e762a64c5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:50.057419 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1145/2280] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:21:49 2022] DELETE /identity/v3/projects/8ef89153c2d24ff99040819c56e54e8c => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:50.059052 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a7eb82-2c47-4874-bb38-2ce65345f4d9 service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:50.073121 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a7eb82-2c47-4874-bb38-2ce65345f4d9 service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:50.172864 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c60309-c2bf-43d6-8b26-392e762a64c5 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:50.174694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-78c60309-c2bf-43d6-8b26-392e762a64c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:50.303015 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630737fe-6f7c-434e-8041-bc8425bb413e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:50.303395 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630737fe-6f7c-434e-8041-bc8425bb413e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:50.303682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630737fe-6f7c-434e-8041-bc8425bb413e admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:50.304047 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630737fe-6f7c-434e-8041-bc8425bb413e admin admin] PATH_INFO: `/v3/users/287ad17db4534a65b3b0850a2ba41117` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:50.305371 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-630737fe-6f7c-434e-8041-bc8425bb413e admin admin] RBAC: Authorizing `identity:delete_user(user_id=287ad17db4534a65b3b0850a2ba41117)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:50.307295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-630737fe-6f7c-434e-8041-bc8425bb413e admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:50.357572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c60309-c2bf-43d6-8b26-392e762a64c5 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:50.368353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf15611a-3294-4d3f-a106-29baa181d3cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:50.369056 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf15611a-3294-4d3f-a106-29baa181d3cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:50.369317 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf15611a-3294-4d3f-a106-29baa181d3cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:50.369615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf15611a-3294-4d3f-a106-29baa181d3cd admin admin] PATH_INFO: `/v3/projects/cd836c63e9f14336a59ea9d51b05510b` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:50.370888 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf15611a-3294-4d3f-a106-29baa181d3cd admin admin] RBAC: Authorizing `identity:delete_project(project_id=cd836c63e9f14336a59ea9d51b05510b)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:50.380444 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf15611a-3294-4d3f-a106-29baa181d3cd admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:50.397375 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-630737fe-6f7c-434e-8041-bc8425bb413e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '287ad17db4534a65b3b0850a2ba41117'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:50.426172 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-630737fe-6f7c-434e-8041-bc8425bb413e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '287ad17db4534a65b3b0850a2ba41117'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:50.426976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-630737fe-6f7c-434e-8041-bc8425bb413e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '287ad17db4534a65b3b0850a2ba41117'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:50.441858 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-630737fe-6f7c-434e-8041-bc8425bb413e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '287ad17db4534a65b3b0850a2ba41117'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:50.472904 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-630737fe-6f7c-434e-8041-bc8425bb413e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '287ad17db4534a65b3b0850a2ba41117'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:50.480787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1136/2281] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:21:50 2022] DELETE /identity/v3/users/287ad17db4534a65b3b0850a2ba41117 => generated 0 bytes in 422 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:21:50.533984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-bf15611a-3294-4d3f-a106-29baa181d3cd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cd836c63e9f14336a59ea9d51b05510b', 'request_id': 'req-bf15611a-3294-4d3f-a106-29baa181d3cd'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:50.541247 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:50.542425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-bf15611a-3294-4d3f-a106-29baa181d3cd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cd836c63e9f14336a59ea9d51b05510b', 'request_id': 'req-bf15611a-3294-4d3f-a106-29baa181d3cd'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:50.553849 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-bf15611a-3294-4d3f-a106-29baa181d3cd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cd836c63e9f14336a59ea9d51b05510b', 'request_id': 'req-bf15611a-3294-4d3f-a106-29baa181d3cd'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:50.572805 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-bf15611a-3294-4d3f-a106-29baa181d3cd admin admin] The token cache is being invalidate because project cd836c63e9f14336a59ea9d51b05510b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:21:50.573354 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-bf15611a-3294-4d3f-a106-29baa181d3cd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:50.574206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-bf15611a-3294-4d3f-a106-29baa181d3cd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:50.576620 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1146/2282] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:21:50 2022] DELETE /identity/v3/projects/cd836c63e9f14336a59ea9d51b05510b => generated 0 bytes in 404 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:50.582564 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7756b5f-fc2e-4eca-b4ba-a8796e75b93e None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:50.582855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7756b5f-fc2e-4eca-b4ba-a8796e75b93e None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:50.583136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7756b5f-fc2e-4eca-b4ba-a8796e75b93e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:50.634611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-b7756b5f-fc2e-4eca-b4ba-a8796e75b93e None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:50.636209 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-b7756b5f-fc2e-4eca-b4ba-a8796e75b93e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:50.722606 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:50.768314 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1137/2283] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:50.774978 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04764e5c-2325-415d-83f2-6cb8b5374009 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:50.775550 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04764e5c-2325-415d-83f2-6cb8b5374009 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:50.776055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04764e5c-2325-415d-83f2-6cb8b5374009 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:50.821802 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-04764e5c-2325-415d-83f2-6cb8b5374009 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:50.829481 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-04764e5c-2325-415d-83f2-6cb8b5374009 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:50.853866 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1147/2284] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:50.856455 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7756b5f-fc2e-4eca-b4ba-a8796e75b93e None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:50.869271 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7756b5f-fc2e-4eca-b4ba-a8796e75b93e None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:50.875819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fad1bcb-fe90-4c49-a741-4e4054733d9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:50.878559 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fad1bcb-fe90-4c49-a741-4e4054733d9b admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:50.878842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fad1bcb-fe90-4c49-a741-4e4054733d9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:50.879122 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fad1bcb-fe90-4c49-a741-4e4054733d9b admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:50.880499 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fad1bcb-fe90-4c49-a741-4e4054733d9b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:21:50.880878 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fad1bcb-fe90-4c49-a741-4e4054733d9b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:50.882879 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fad1bcb-fe90-4c49-a741-4e4054733d9b admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:50.904486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1138/2285] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:21:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:21:50.905925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04764e5c-2325-415d-83f2-6cb8b5374009 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:50.928002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04764e5c-2325-415d-83f2-6cb8b5374009 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:50.941965 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716321f4-7fdc-4336-9a4f-f7affcf10962 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:50.942878 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716321f4-7fdc-4336-9a4f-f7affcf10962 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:50.943466 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716321f4-7fdc-4336-9a4f-f7affcf10962 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:50.943914 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716321f4-7fdc-4336-9a4f-f7affcf10962 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:50.944713 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-716321f4-7fdc-4336-9a4f-f7affcf10962 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:50.946954 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-716321f4-7fdc-4336-9a4f-f7affcf10962 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:51.017196 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1148/2286] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:21:50 2022] POST /identity/v3/projects => generated 365 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:21:51.021772 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fad1bcb-fe90-4c49-a741-4e4054733d9b admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:51.034093 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fad1bcb-fe90-4c49-a741-4e4054733d9b admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:51.143163 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ccea73-8f01-44b7-83a6-0e7128d7b953 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:51.143854 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ccea73-8f01-44b7-83a6-0e7128d7b953 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:51.144253 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ccea73-8f01-44b7-83a6-0e7128d7b953 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:51.144587 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ccea73-8f01-44b7-83a6-0e7128d7b953 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:51.145662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ccea73-8f01-44b7-83a6-0e7128d7b953 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:51.148155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ccea73-8f01-44b7-83a6-0e7128d7b953 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:51.183043 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1139/2287] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:21:51 2022] POST /identity/v3/users => generated 283 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:21:51.187409 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716321f4-7fdc-4336-9a4f-f7affcf10962 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:51.212519 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716321f4-7fdc-4336-9a4f-f7affcf10962 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:51.224689 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b9f4f3-3497-4ff2-a3b1-45147ed8111f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:51.225367 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b9f4f3-3497-4ff2-a3b1-45147ed8111f admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:51.225769 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b9f4f3-3497-4ff2-a3b1-45147ed8111f admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:51.226144 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b9f4f3-3497-4ff2-a3b1-45147ed8111f admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:51.226917 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b9f4f3-3497-4ff2-a3b1-45147ed8111f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:51.229290 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b9f4f3-3497-4ff2-a3b1-45147ed8111f admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:51.258327 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1149/2288] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:51 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:21:51.258569 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ccea73-8f01-44b7-83a6-0e7128d7b953 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:51.278982 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ccea73-8f01-44b7-83a6-0e7128d7b953 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:51.294456 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191b97f4-d40b-4101-9386-98ba3cd314d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:51.295135 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191b97f4-d40b-4101-9386-98ba3cd314d0 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:51.295453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191b97f4-d40b-4101-9386-98ba3cd314d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:51.296315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191b97f4-d40b-4101-9386-98ba3cd314d0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:51.300235 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191b97f4-d40b-4101-9386-98ba3cd314d0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:51.307580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191b97f4-d40b-4101-9386-98ba3cd314d0 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:51.331264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1140/2289] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:51 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:21:51.335086 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b9f4f3-3497-4ff2-a3b1-45147ed8111f admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:51.378508 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b9f4f3-3497-4ff2-a3b1-45147ed8111f admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:51.393765 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4025b8c6-bcac-4414-bd1a-e09400ca7834 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:51.394657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4025b8c6-bcac-4414-bd1a-e09400ca7834 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:51.395032 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4025b8c6-bcac-4414-bd1a-e09400ca7834 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:51.395409 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4025b8c6-bcac-4414-bd1a-e09400ca7834 admin admin] PATH_INFO: `/v3/projects/a613cbeb387243ccb4d167d82c5068de/users/7c13d3c029be4a56a6395521bc6b4bb7/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:51.460029 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4025b8c6-bcac-4414-bd1a-e09400ca7834 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a613cbeb387243ccb4d167d82c5068de, user_id=7c13d3c029be4a56a6395521bc6b4bb7, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:51.466194 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4025b8c6-bcac-4414-bd1a-e09400ca7834 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:51.518434 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1150/2290] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:21:51 2022] PUT /identity/v3/projects/a613cbeb387243ccb4d167d82c5068de/users/7c13d3c029be4a56a6395521bc6b4bb7/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:51.533979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce19bb4f-638a-4b44-b965-602187ee1c61 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:51.535047 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce19bb4f-638a-4b44-b965-602187ee1c61 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:51.536403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce19bb4f-638a-4b44-b965-602187ee1c61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:51.587550 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-ce19bb4f-638a-4b44-b965-602187ee1c61 None None] MFA Rules not processed for user `7c13d3c029be4a56a6395521bc6b4bb7`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:51.590445 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ce19bb4f-638a-4b44-b965-602187ee1c61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:51.720225 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1141/2291] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:51 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:51.734480 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4025b8c6-bcac-4414-bd1a-e09400ca7834 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:51.736725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-4025b8c6-bcac-4414-bd1a-e09400ca7834 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:51.923747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4025b8c6-bcac-4414-bd1a-e09400ca7834 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:51.940988 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b633d1a2-e1a4-4207-9b0c-8ffa1deb6703 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:51.943621 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b633d1a2-e1a4-4207-9b0c-8ffa1deb6703 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:51.943933 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b633d1a2-e1a4-4207-9b0c-8ffa1deb6703 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:51.944844 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b633d1a2-e1a4-4207-9b0c-8ffa1deb6703 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:51.965143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce19bb4f-638a-4b44-b965-602187ee1c61 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:51.966849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ce19bb4f-638a-4b44-b965-602187ee1c61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:51.971107 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b633d1a2-e1a4-4207-9b0c-8ffa1deb6703 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:51.973258 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b633d1a2-e1a4-4207-9b0c-8ffa1deb6703 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:52.003123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1151/2292] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:51 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 270 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:21:52.137384 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:52.216575 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce19bb4f-638a-4b44-b965-602187ee1c61 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:52.240476 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51f7e61-c79a-4cae-9eb4-1e975a1194c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:52.241548 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51f7e61-c79a-4cae-9eb4-1e975a1194c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:52.242119 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51f7e61-c79a-4cae-9eb4-1e975a1194c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:52.242471 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51f7e61-c79a-4cae-9eb4-1e975a1194c9 admin admin] PATH_INFO: `/v3/users/887a8cde38b144709b9ded8adcaeedbe` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:52.246880 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a51f7e61-c79a-4cae-9eb4-1e975a1194c9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=887a8cde38b144709b9ded8adcaeedbe)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:52.249630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a51f7e61-c79a-4cae-9eb4-1e975a1194c9 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:52.451653 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b633d1a2-e1a4-4207-9b0c-8ffa1deb6703 service cinder] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:52.453473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-b633d1a2-e1a4-4207-9b0c-8ffa1deb6703 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:52.463122 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a51f7e61-c79a-4cae-9eb4-1e975a1194c9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '887a8cde38b144709b9ded8adcaeedbe'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:52.501247 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b633d1a2-e1a4-4207-9b0c-8ffa1deb6703 service cinder] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:52.502085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a51f7e61-c79a-4cae-9eb4-1e975a1194c9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '887a8cde38b144709b9ded8adcaeedbe'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:52.509598 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a51f7e61-c79a-4cae-9eb4-1e975a1194c9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '887a8cde38b144709b9ded8adcaeedbe'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:52.525344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d71cae3-2c14-4693-a6c0-1a22d164420d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:52.527559 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d71cae3-2c14-4693-a6c0-1a22d164420d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:52.529493 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d71cae3-2c14-4693-a6c0-1a22d164420d admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:52.531127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d71cae3-2c14-4693-a6c0-1a22d164420d admin admin] PATH_INFO: `/v3/projects/7de4e6ebc0554132a6ad5ee20c9c44d6` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:52.536534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d71cae3-2c14-4693-a6c0-1a22d164420d admin admin] RBAC: Authorizing `identity:delete_project(project_id=7de4e6ebc0554132a6ad5ee20c9c44d6)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:52.539367 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d71cae3-2c14-4693-a6c0-1a22d164420d admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:52.541117 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a51f7e61-c79a-4cae-9eb4-1e975a1194c9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '887a8cde38b144709b9ded8adcaeedbe'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:52.567142 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a51f7e61-c79a-4cae-9eb4-1e975a1194c9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '887a8cde38b144709b9ded8adcaeedbe'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:52.581631 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1142/2293] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:21:51 2022] DELETE /identity/v3/users/887a8cde38b144709b9ded8adcaeedbe => generated 0 bytes in 618 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:52.587061 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5533490-11e1-49eb-ac71-4b6716e7ead0 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:52.588042 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5533490-11e1-49eb-ac71-4b6716e7ead0 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:52.588487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5533490-11e1-49eb-ac71-4b6716e7ead0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:52.623065 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-f5533490-11e1-49eb-ac71-4b6716e7ead0 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:52.624868 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-f5533490-11e1-49eb-ac71-4b6716e7ead0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:52.671215 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-4d71cae3-2c14-4693-a6c0-1a22d164420d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7de4e6ebc0554132a6ad5ee20c9c44d6', 'request_id': 'req-4d71cae3-2c14-4693-a6c0-1a22d164420d'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:52.682781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-4d71cae3-2c14-4693-a6c0-1a22d164420d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7de4e6ebc0554132a6ad5ee20c9c44d6', 'request_id': 'req-4d71cae3-2c14-4693-a6c0-1a22d164420d'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:52.694877 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-4d71cae3-2c14-4693-a6c0-1a22d164420d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7de4e6ebc0554132a6ad5ee20c9c44d6', 'request_id': 'req-4d71cae3-2c14-4693-a6c0-1a22d164420d'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:52.722627 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-4d71cae3-2c14-4693-a6c0-1a22d164420d admin admin] The token cache is being invalidate because project 7de4e6ebc0554132a6ad5ee20c9c44d6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:21:52.723752 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-4d71cae3-2c14-4693-a6c0-1a22d164420d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:52.725660 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-4d71cae3-2c14-4693-a6c0-1a22d164420d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:52.728756 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1152/2294] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:21:52 2022] DELETE /identity/v3/projects/7de4e6ebc0554132a6ad5ee20c9c44d6 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:21:52.732262 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d71cae3-2c14-4693-a6c0-1a22d164420d admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:52.738510 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-4d71cae3-2c14-4693-a6c0-1a22d164420d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:52.758282 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:DELETE=9,keystone:SELECT=87,keystone:INSERT=4 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:52.811511 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1143/2295] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:21:52.844387 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5533490-11e1-49eb-ac71-4b6716e7ead0 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:52.846091 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-f5533490-11e1-49eb-ac71-4b6716e7ead0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:52.936965 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d71cae3-2c14-4693-a6c0-1a22d164420d admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:52.955246 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b3e778-f988-4a1b-8041-39a6fdcb0d57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:52.955971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b3e778-f988-4a1b-8041-39a6fdcb0d57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:52.956228 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b3e778-f988-4a1b-8041-39a6fdcb0d57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:52.956499 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b3e778-f988-4a1b-8041-39a6fdcb0d57 admin admin] PATH_INFO: `/v3/users/e19b722cf8d049fa83d8f18ae67ff664` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:52.958334 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b3e778-f988-4a1b-8041-39a6fdcb0d57 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e19b722cf8d049fa83d8f18ae67ff664)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:52.960609 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b3e778-f988-4a1b-8041-39a6fdcb0d57 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:52.976824 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5533490-11e1-49eb-ac71-4b6716e7ead0 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:52.981638 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cb4457-04da-42ff-909c-b5d2318c68bc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:52.982317 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cb4457-04da-42ff-909c-b5d2318c68bc service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:52.982599 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cb4457-04da-42ff-909c-b5d2318c68bc service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:52.982857 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cb4457-04da-42ff-909c-b5d2318c68bc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:52.994330 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97cb4457-04da-42ff-909c-b5d2318c68bc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:52.996015 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97cb4457-04da-42ff-909c-b5d2318c68bc service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:53.003303 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:53.021051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1144/2296] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:21:53.080694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-79b3e778-f988-4a1b-8041-39a6fdcb0d57 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e19b722cf8d049fa83d8f18ae67ff664'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:53.102674 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-79b3e778-f988-4a1b-8041-39a6fdcb0d57 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e19b722cf8d049fa83d8f18ae67ff664'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:53.104759 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-79b3e778-f988-4a1b-8041-39a6fdcb0d57 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e19b722cf8d049fa83d8f18ae67ff664'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:53.120470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-79b3e778-f988-4a1b-8041-39a6fdcb0d57 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e19b722cf8d049fa83d8f18ae67ff664'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:53.160983 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-79b3e778-f988-4a1b-8041-39a6fdcb0d57 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e19b722cf8d049fa83d8f18ae67ff664'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:53.176380 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1153/2297] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:21:52 2022] DELETE /identity/v3/users/e19b722cf8d049fa83d8f18ae67ff664 => generated 0 bytes in 444 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:53.870542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cb4457-04da-42ff-909c-b5d2318c68bc service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:53.872712 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-97cb4457-04da-42ff-909c-b5d2318c68bc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:53.894627 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cb4457-04da-42ff-909c-b5d2318c68bc service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:54.093430 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9aa7014-ff33-4317-abdc-baa98a363b6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:54.094621 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9aa7014-ff33-4317-abdc-baa98a363b6f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:54.095031 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9aa7014-ff33-4317-abdc-baa98a363b6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:54.095856 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9aa7014-ff33-4317-abdc-baa98a363b6f admin admin] PATH_INFO: `/v3/projects/5ada2c71f3c948f58cec85e4edaf1a92` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:54.098361 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9aa7014-ff33-4317-abdc-baa98a363b6f admin admin] RBAC: Authorizing `identity:delete_project(project_id=5ada2c71f3c948f58cec85e4edaf1a92)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:54.102202 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9aa7014-ff33-4317-abdc-baa98a363b6f admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:54.123357 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b3e778-f988-4a1b-8041-39a6fdcb0d57 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:54.124944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-79b3e778-f988-4a1b-8041-39a6fdcb0d57 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:54.142210 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b3e778-f988-4a1b-8041-39a6fdcb0d57 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:54.165186 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423b7aea-241e-4b89-af09-a80ff5f52615 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:54.166583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423b7aea-241e-4b89-af09-a80ff5f52615 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:54.167217 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423b7aea-241e-4b89-af09-a80ff5f52615 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:54.167648 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423b7aea-241e-4b89-af09-a80ff5f52615 admin admin] PATH_INFO: `/v3/projects/cb383707cc014e58ad6685429e2f3388` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:54.169417 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423b7aea-241e-4b89-af09-a80ff5f52615 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cb383707cc014e58ad6685429e2f3388)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:54.174522 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423b7aea-241e-4b89-af09-a80ff5f52615 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:54.261144 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-e9aa7014-ff33-4317-abdc-baa98a363b6f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5ada2c71f3c948f58cec85e4edaf1a92', 'request_id': 'req-e9aa7014-ff33-4317-abdc-baa98a363b6f'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:54.269717 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-e9aa7014-ff33-4317-abdc-baa98a363b6f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5ada2c71f3c948f58cec85e4edaf1a92', 'request_id': 'req-e9aa7014-ff33-4317-abdc-baa98a363b6f'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:54.293508 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-423b7aea-241e-4b89-af09-a80ff5f52615 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cb383707cc014e58ad6685429e2f3388', 'request_id': 'req-423b7aea-241e-4b89-af09-a80ff5f52615'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:54.297121 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-e9aa7014-ff33-4317-abdc-baa98a363b6f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5ada2c71f3c948f58cec85e4edaf1a92', 'request_id': 'req-e9aa7014-ff33-4317-abdc-baa98a363b6f'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:54.304612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-423b7aea-241e-4b89-af09-a80ff5f52615 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cb383707cc014e58ad6685429e2f3388', 'request_id': 'req-423b7aea-241e-4b89-af09-a80ff5f52615'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:54.315791 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-423b7aea-241e-4b89-af09-a80ff5f52615 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cb383707cc014e58ad6685429e2f3388', 'request_id': 'req-423b7aea-241e-4b89-af09-a80ff5f52615'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:54.317392 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-e9aa7014-ff33-4317-abdc-baa98a363b6f admin admin] The token cache is being invalidate because project 5ada2c71f3c948f58cec85e4edaf1a92 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:21:54.318216 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-e9aa7014-ff33-4317-abdc-baa98a363b6f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:54.319301 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-e9aa7014-ff33-4317-abdc-baa98a363b6f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:54.322279 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1145/2298] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:21:53 2022] DELETE /identity/v3/projects/5ada2c71f3c948f58cec85e4edaf1a92 => generated 0 bytes in 453 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:54.328806 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96910634-1486-4185-a1e5-1f03659bd468 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:54.329781 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96910634-1486-4185-a1e5-1f03659bd468 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:54.331477 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96910634-1486-4185-a1e5-1f03659bd468 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:54.341259 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-423b7aea-241e-4b89-af09-a80ff5f52615 admin admin] The token cache is being invalidate because project cb383707cc014e58ad6685429e2f3388 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:21:54.343447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-423b7aea-241e-4b89-af09-a80ff5f52615 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:54.345545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-423b7aea-241e-4b89-af09-a80ff5f52615 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:21:54.354049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1154/2299] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:21:54 2022] DELETE /identity/v3/projects/cb383707cc014e58ad6685429e2f3388 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:21:54.367546 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-96910634-1486-4185-a1e5-1f03659bd468 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:54.369436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-96910634-1486-4185-a1e5-1f03659bd468 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:54.528158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1146/2300] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:54.545249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31407451-2150-4ba8-8d7a-867d510afa4e None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:54.546704 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31407451-2150-4ba8-8d7a-867d510afa4e None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:54.547188 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31407451-2150-4ba8-8d7a-867d510afa4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:54.574120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:54.586321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-31407451-2150-4ba8-8d7a-867d510afa4e None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:54.591287 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-31407451-2150-4ba8-8d7a-867d510afa4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:54.622733 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1155/2301] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:54.626809 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96910634-1486-4185-a1e5-1f03659bd468 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:54.640948 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96910634-1486-4185-a1e5-1f03659bd468 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:54.654774 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf3018a-76f9-46ff-a3ed-5c0645bd565c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:54.656334 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf3018a-76f9-46ff-a3ed-5c0645bd565c admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:54.656713 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf3018a-76f9-46ff-a3ed-5c0645bd565c admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:54.657088 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf3018a-76f9-46ff-a3ed-5c0645bd565c admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:54.664732 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf3018a-76f9-46ff-a3ed-5c0645bd565c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:21:54.665179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf3018a-76f9-46ff-a3ed-5c0645bd565c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:54.667047 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf3018a-76f9-46ff-a3ed-5c0645bd565c admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:54.702067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1147/2302] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:21:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:21:54.703586 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31407451-2150-4ba8-8d7a-867d510afa4e None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:54.717137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31407451-2150-4ba8-8d7a-867d510afa4e None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:54.732335 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab5fff8-47ba-481a-ba4e-75fffa904ad9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:54.733237 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab5fff8-47ba-481a-ba4e-75fffa904ad9 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:54.734221 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab5fff8-47ba-481a-ba4e-75fffa904ad9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:54.734601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab5fff8-47ba-481a-ba4e-75fffa904ad9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:54.736527 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab5fff8-47ba-481a-ba4e-75fffa904ad9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:54.741396 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab5fff8-47ba-481a-ba4e-75fffa904ad9 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:54.805848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1156/2303] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:21:54 2022] POST /identity/v3/projects => generated 381 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:21:54.819352 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf3018a-76f9-46ff-a3ed-5c0645bd565c admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:54.842074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf3018a-76f9-46ff-a3ed-5c0645bd565c admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:54.873257 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:21:54.988516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b044558e-5cd6-4625-91d8-caa7ab6bf265 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:54.988516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b044558e-5cd6-4625-91d8-caa7ab6bf265 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:54.988516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b044558e-5cd6-4625-91d8-caa7ab6bf265 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:54.988516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b044558e-5cd6-4625-91d8-caa7ab6bf265 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:54.989286 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b044558e-5cd6-4625-91d8-caa7ab6bf265 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:54.989286 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b044558e-5cd6-4625-91d8-caa7ab6bf265 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:55.036479 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1148/2304] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:21:54 2022] POST /identity/v3/users => generated 291 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:21:55.040451 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab5fff8-47ba-481a-ba4e-75fffa904ad9 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:55.056088 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab5fff8-47ba-481a-ba4e-75fffa904ad9 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:55.067572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8876f390-e911-4340-8a9c-2991491d197e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:55.068273 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8876f390-e911-4340-8a9c-2991491d197e admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:55.068770 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8876f390-e911-4340-8a9c-2991491d197e admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:55.070601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8876f390-e911-4340-8a9c-2991491d197e admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:55.071494 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8876f390-e911-4340-8a9c-2991491d197e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:55.074063 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8876f390-e911-4340-8a9c-2991491d197e admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:55.097792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1157/2305] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:55 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:21:55.099469 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b044558e-5cd6-4625-91d8-caa7ab6bf265 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:55.120413 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b044558e-5cd6-4625-91d8-caa7ab6bf265 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:55.126965 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92394a5e-e985-4669-81a2-817c91f4ef39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:55.127181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92394a5e-e985-4669-81a2-817c91f4ef39 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:55.128308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92394a5e-e985-4669-81a2-817c91f4ef39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:55.128308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92394a5e-e985-4669-81a2-817c91f4ef39 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:55.129042 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92394a5e-e985-4669-81a2-817c91f4ef39 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:55.131524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92394a5e-e985-4669-81a2-817c91f4ef39 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:55.163832 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1149/2306] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:21:55 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:21:55.169400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8876f390-e911-4340-8a9c-2991491d197e admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:55.189049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8876f390-e911-4340-8a9c-2991491d197e admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:55.194999 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37931444-dc8c-4e69-afcd-9cbe8952049c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:55.195674 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37931444-dc8c-4e69-afcd-9cbe8952049c admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:55.195958 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37931444-dc8c-4e69-afcd-9cbe8952049c admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:55.196270 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37931444-dc8c-4e69-afcd-9cbe8952049c admin admin] PATH_INFO: `/v3/projects/14d42cf78c32468cb8bf10aa847c7cc3/users/782ebdc88e21483eb3cecd00415b9b07/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:55.223825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37931444-dc8c-4e69-afcd-9cbe8952049c admin admin] RBAC: Authorizing `identity:create_grant(project_id=14d42cf78c32468cb8bf10aa847c7cc3, user_id=782ebdc88e21483eb3cecd00415b9b07, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:55.227044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37931444-dc8c-4e69-afcd-9cbe8952049c admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:55.254766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1158/2307] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:21:55 2022] PUT /identity/v3/projects/14d42cf78c32468cb8bf10aa847c7cc3/users/782ebdc88e21483eb3cecd00415b9b07/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:21:55.260408 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92f89f4-c170-4c63-a19c-5b7947da74ae None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:55.260711 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92f89f4-c170-4c63-a19c-5b7947da74ae None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:55.262333 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92f89f4-c170-4c63-a19c-5b7947da74ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:55.312350 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-e92f89f4-c170-4c63-a19c-5b7947da74ae None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:55.318809 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-e92f89f4-c170-4c63-a19c-5b7947da74ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:55.467827 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1150/2308] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:55.477588 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37931444-dc8c-4e69-afcd-9cbe8952049c admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:55.479745 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-37931444-dc8c-4e69-afcd-9cbe8952049c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:55.562804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37931444-dc8c-4e69-afcd-9cbe8952049c admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:55.566536 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bacfab4-7cf7-4a02-880d-23f2af86e03a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:55.567135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bacfab4-7cf7-4a02-880d-23f2af86e03a service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:55.567409 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bacfab4-7cf7-4a02-880d-23f2af86e03a service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:55.567795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bacfab4-7cf7-4a02-880d-23f2af86e03a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:55.578068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bacfab4-7cf7-4a02-880d-23f2af86e03a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:55.579511 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bacfab4-7cf7-4a02-880d-23f2af86e03a service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:55.588872 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1159/2309] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:21:56.910739 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160a99d6-ae5a-4459-a0d1-41e3c9c3bc79 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:56.911302 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160a99d6-ae5a-4459-a0d1-41e3c9c3bc79 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:56.911302 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160a99d6-ae5a-4459-a0d1-41e3c9c3bc79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:56.967298 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-160a99d6-ae5a-4459-a0d1-41e3c9c3bc79 None None] MFA Rules not processed for user `782ebdc88e21483eb3cecd00415b9b07`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:56.969219 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-160a99d6-ae5a-4459-a0d1-41e3c9c3bc79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:57.059522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1151/2310] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:56 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:21:57.071079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bacfab4-7cf7-4a02-880d-23f2af86e03a service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:21:57.073407 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-2bacfab4-7cf7-4a02-880d-23f2af86e03a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:57.160569 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bacfab4-7cf7-4a02-880d-23f2af86e03a service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:21:57.164924 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db60de4-f392-40e0-a9a4-a31c208797ba service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:21:57.165850 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db60de4-f392-40e0-a9a4-a31c208797ba service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:57.166135 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db60de4-f392-40e0-a9a4-a31c208797ba service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:57.166479 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db60de4-f392-40e0-a9a4-a31c208797ba service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:57.175424 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db60de4-f392-40e0-a9a4-a31c208797ba service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:21:57.176846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db60de4-f392-40e0-a9a4-a31c208797ba service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:21:57.185691 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1160/2311] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:21:57 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:21:58.688418 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3108be-a1f0-4d5d-99c5-b0e98c29fef7 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:21:58.688889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3108be-a1f0-4d5d-99c5-b0e98c29fef7 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:21:58.689096 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3108be-a1f0-4d5d-99c5-b0e98c29fef7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:21:58.708082 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-1f3108be-a1f0-4d5d-99c5-b0e98c29fef7 None None] MFA Rules not processed for user `782ebdc88e21483eb3cecd00415b9b07`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:21:58.709392 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-1f3108be-a1f0-4d5d-99c5-b0e98c29fef7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:21:58.716238 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1152/2312] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:21:58 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:22:07.181641 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=76,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:22:08.702180 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:22:09.720233 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a055b689-d1be-42bb-aa4b-6ce7f1fc6ed1 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:09.720733 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a055b689-d1be-42bb-aa4b-6ce7f1fc6ed1 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:09.720834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a055b689-d1be-42bb-aa4b-6ce7f1fc6ed1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:09.757390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-a055b689-d1be-42bb-aa4b-6ce7f1fc6ed1 None None] MFA Rules not processed for user `5a2fcc8c9ac44dceadcc30d163bbfa0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:22:09.761065 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a055b689-d1be-42bb-aa4b-6ce7f1fc6ed1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:09.914959 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1161/2313] 10.222.0.26 () {58 vars in 1009 bytes} [Tue Sep 27 04:22:09 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:22:09.931661 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3108be-a1f0-4d5d-99c5-b0e98c29fef7 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:09.934285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-1f3108be-a1f0-4d5d-99c5-b0e98c29fef7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:10.073311 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3108be-a1f0-4d5d-99c5-b0e98c29fef7 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:10.079261 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7e7941-6b29-4317-8122-107ef183ac8e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '181dd5dc720c4a97860852062b7b19c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:10.093775 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7e7941-6b29-4317-8122-107ef183ac8e service swift] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:10.093775 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7e7941-6b29-4317-8122-107ef183ac8e service swift] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:10.093775 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7e7941-6b29-4317-8122-107ef183ac8e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:10.111979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7e7941-6b29-4317-8122-107ef183ac8e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:10.113805 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7e7941-6b29-4317-8122-107ef183ac8e service swift] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:10.128045 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1153/2314] 10.222.0.26 () {60 vars in 1384 bytes} [Tue Sep 27 04:22:09 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 04:22:10.829108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c658b284-b738-422a-a2e3-bc9114d31d19 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:10.829627 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c658b284-b738-422a-a2e3-bc9114d31d19 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:10.829757 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c658b284-b738-422a-a2e3-bc9114d31d19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:10.849186 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-c658b284-b738-422a-a2e3-bc9114d31d19 None None] MFA Rules not processed for user `5a2fcc8c9ac44dceadcc30d163bbfa0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:22:10.850475 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c658b284-b738-422a-a2e3-bc9114d31d19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:10.857703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1162/2315] 10.222.0.26 () {58 vars in 967 bytes} [Tue Sep 27 04:22:10 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:22:10.870315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7e7941-6b29-4317-8122-107ef183ac8e service swift] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:10.880945 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7e7941-6b29-4317-8122-107ef183ac8e service swift] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:10.885994 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c3b89c-f38a-4b0c-afb5-4244f5553c89 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '181dd5dc720c4a97860852062b7b19c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:10.886576 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c3b89c-f38a-4b0c-afb5-4244f5553c89 service swift] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:10.886862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c3b89c-f38a-4b0c-afb5-4244f5553c89 service swift] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:10.887138 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c3b89c-f38a-4b0c-afb5-4244f5553c89 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:10.895827 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c3b89c-f38a-4b0c-afb5-4244f5553c89 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:10.897585 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c3b89c-f38a-4b0c-afb5-4244f5553c89 service swift] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:10.906934 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1154/2316] 10.222.0.26 () {60 vars in 1384 bytes} [Tue Sep 27 04:22:10 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 04:22:11.174384 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c301446-1111-4557-9798-59f08fd1579f None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:11.174857 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c301446-1111-4557-9798-59f08fd1579f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:11.175248 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c301446-1111-4557-9798-59f08fd1579f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:11.198083 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-1c301446-1111-4557-9798-59f08fd1579f None None] MFA Rules not processed for user `5a2fcc8c9ac44dceadcc30d163bbfa0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:22:11.199301 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-1c301446-1111-4557-9798-59f08fd1579f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:11.207396 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1163/2317] 10.222.0.26 () {58 vars in 1009 bytes} [Tue Sep 27 04:22:11 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:22:11.221457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c3b89c-f38a-4b0c-afb5-4244f5553c89 service swift] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:11.232904 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c3b89c-f38a-4b0c-afb5-4244f5553c89 service swift] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:11.241650 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05df66f3-f372-4a3e-9dd7-450a8a39b867 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '181dd5dc720c4a97860852062b7b19c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:11.242594 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05df66f3-f372-4a3e-9dd7-450a8a39b867 service swift] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:11.243037 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05df66f3-f372-4a3e-9dd7-450a8a39b867 service swift] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:11.243472 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05df66f3-f372-4a3e-9dd7-450a8a39b867 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:11.254054 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05df66f3-f372-4a3e-9dd7-450a8a39b867 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:11.255959 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05df66f3-f372-4a3e-9dd7-450a8a39b867 service swift] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:11.275256 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1155/2318] 10.222.0.26 () {60 vars in 1384 bytes} [Tue Sep 27 04:22:11 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 04:22:11.631904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d83acb-6ece-47d5-8858-fa95f0f67347 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:11.632182 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d83acb-6ece-47d5-8858-fa95f0f67347 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:11.632456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d83acb-6ece-47d5-8858-fa95f0f67347 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:11.655269 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-62d83acb-6ece-47d5-8858-fa95f0f67347 None None] MFA Rules not processed for user `5a2fcc8c9ac44dceadcc30d163bbfa0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:22:11.656838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-62d83acb-6ece-47d5-8858-fa95f0f67347 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:11.665167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1164/2319] 10.222.0.26 () {58 vars in 967 bytes} [Tue Sep 27 04:22:11 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:22:11.681361 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05df66f3-f372-4a3e-9dd7-450a8a39b867 service swift] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:11.708756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05df66f3-f372-4a3e-9dd7-450a8a39b867 service swift] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:11.714398 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b03d48-58e6-4c6d-bfe2-25108a91b4d8 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '181dd5dc720c4a97860852062b7b19c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:11.715086 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b03d48-58e6-4c6d-bfe2-25108a91b4d8 service swift] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:11.715450 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b03d48-58e6-4c6d-bfe2-25108a91b4d8 service swift] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:11.716883 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b03d48-58e6-4c6d-bfe2-25108a91b4d8 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:11.726686 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b03d48-58e6-4c6d-bfe2-25108a91b4d8 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:11.728364 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b03d48-58e6-4c6d-bfe2-25108a91b4d8 service swift] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:11.739798 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1156/2320] 10.222.0.26 () {60 vars in 1384 bytes} [Tue Sep 27 04:22:11 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 27 04:22:14.034110 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d83acb-6ece-47d5-8858-fa95f0f67347 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:14.035656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-62d83acb-6ece-47d5-8858-fa95f0f67347 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:14.054538 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d83acb-6ece-47d5-8858-fa95f0f67347 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:14.064955 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0226116b-928c-4c63-a92e-647ef453ca13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:14.065680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0226116b-928c-4c63-a92e-647ef453ca13 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:14.066827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0226116b-928c-4c63-a92e-647ef453ca13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:14.067205 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0226116b-928c-4c63-a92e-647ef453ca13 admin admin] PATH_INFO: `/v3/users/ad63c13d1c284221a761ff3550b4c6ca` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:14.068600 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0226116b-928c-4c63-a92e-647ef453ca13 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ad63c13d1c284221a761ff3550b4c6ca)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:14.070759 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0226116b-928c-4c63-a92e-647ef453ca13 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:14.137893 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0226116b-928c-4c63-a92e-647ef453ca13 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ad63c13d1c284221a761ff3550b4c6ca'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:14.147972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0226116b-928c-4c63-a92e-647ef453ca13 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ad63c13d1c284221a761ff3550b4c6ca'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:14.148826 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0226116b-928c-4c63-a92e-647ef453ca13 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ad63c13d1c284221a761ff3550b4c6ca'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:14.161432 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0226116b-928c-4c63-a92e-647ef453ca13 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ad63c13d1c284221a761ff3550b4c6ca'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:14.195752 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0226116b-928c-4c63-a92e-647ef453ca13 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ad63c13d1c284221a761ff3550b4c6ca'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:14.211365 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1165/2321] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:22:14 2022] DELETE /identity/v3/users/ad63c13d1c284221a761ff3550b4c6ca => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:22:14.215581 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383635f1-a5ef-48c3-a6b5-993a20d70769 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:14.216102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383635f1-a5ef-48c3-a6b5-993a20d70769 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:14.216213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383635f1-a5ef-48c3-a6b5-993a20d70769 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:14.242187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-383635f1-a5ef-48c3-a6b5-993a20d70769 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:22:14.243634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-383635f1-a5ef-48c3-a6b5-993a20d70769 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:14.343975 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1157/2322] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:22:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:22:14.355347 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0226116b-928c-4c63-a92e-647ef453ca13 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:14.356623 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-0226116b-928c-4c63-a92e-647ef453ca13 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:14.507204 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0226116b-928c-4c63-a92e-647ef453ca13 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:14.512499 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b9ee15-68a9-4f7f-b04c-b28d6598f201 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:14.513503 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b9ee15-68a9-4f7f-b04c-b28d6598f201 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:14.514162 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b9ee15-68a9-4f7f-b04c-b28d6598f201 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:14.514776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b9ee15-68a9-4f7f-b04c-b28d6598f201 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:14.544356 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b9ee15-68a9-4f7f-b04c-b28d6598f201 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:14.546665 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b9ee15-68a9-4f7f-b04c-b28d6598f201 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:14.552970 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:22:14.558924 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1166/2323] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:22:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:22:14.837514 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383635f1-a5ef-48c3-a6b5-993a20d70769 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:14.839373 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-383635f1-a5ef-48c3-a6b5-993a20d70769 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:14.852849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383635f1-a5ef-48c3-a6b5-993a20d70769 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:14.857778 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba581a2-56ae-4b41-b2c9-fa203be8b905 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:14.859112 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba581a2-56ae-4b41-b2c9-fa203be8b905 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:14.859377 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba581a2-56ae-4b41-b2c9-fa203be8b905 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:14.859652 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba581a2-56ae-4b41-b2c9-fa203be8b905 admin admin] PATH_INFO: `/v3/projects/988a2e4b6d154a4da1ee1a1f4d696588` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:14.865055 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba581a2-56ae-4b41-b2c9-fa203be8b905 admin admin] RBAC: Authorizing `identity:delete_project(project_id=988a2e4b6d154a4da1ee1a1f4d696588)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:14.865286 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba581a2-56ae-4b41-b2c9-fa203be8b905 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:14.911838 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:22:14.914977 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-bba581a2-56ae-4b41-b2c9-fa203be8b905 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '988a2e4b6d154a4da1ee1a1f4d696588', 'request_id': 'req-bba581a2-56ae-4b41-b2c9-fa203be8b905'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:14.926548 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-bba581a2-56ae-4b41-b2c9-fa203be8b905 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '988a2e4b6d154a4da1ee1a1f4d696588', 'request_id': 'req-bba581a2-56ae-4b41-b2c9-fa203be8b905'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:14.933448 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-bba581a2-56ae-4b41-b2c9-fa203be8b905 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '988a2e4b6d154a4da1ee1a1f4d696588', 'request_id': 'req-bba581a2-56ae-4b41-b2c9-fa203be8b905'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:14.952145 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-bba581a2-56ae-4b41-b2c9-fa203be8b905 admin admin] The token cache is being invalidate because project 988a2e4b6d154a4da1ee1a1f4d696588 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:22:14.952145 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-bba581a2-56ae-4b41-b2c9-fa203be8b905 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:14.952145 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-bba581a2-56ae-4b41-b2c9-fa203be8b905 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:14.952703 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1158/2324] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:22:14 2022] DELETE /identity/v3/projects/988a2e4b6d154a4da1ee1a1f4d696588 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:22:17.302735 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24233f46-3d7e-4246-86e0-ba4204228528 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:17.303188 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24233f46-3d7e-4246-86e0-ba4204228528 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:17.303292 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24233f46-3d7e-4246-86e0-ba4204228528 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:17.322445 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-24233f46-3d7e-4246-86e0-ba4204228528 None None] MFA Rules not processed for user `5a2fcc8c9ac44dceadcc30d163bbfa0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:22:17.323633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-24233f46-3d7e-4246-86e0-ba4204228528 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:17.379802 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1167/2325] 10.222.0.26 () {58 vars in 1009 bytes} [Tue Sep 27 04:22:17 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:22:17.392517 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba581a2-56ae-4b41-b2c9-fa203be8b905 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:17.393936 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-bba581a2-56ae-4b41-b2c9-fa203be8b905 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:17.487225 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba581a2-56ae-4b41-b2c9-fa203be8b905 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:17.493678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f118d802-9da1-4f43-af70-02146f1ba405 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '181dd5dc720c4a97860852062b7b19c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:17.494322 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f118d802-9da1-4f43-af70-02146f1ba405 service swift] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:17.494682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f118d802-9da1-4f43-af70-02146f1ba405 service swift] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:17.494853 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f118d802-9da1-4f43-af70-02146f1ba405 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:17.504745 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f118d802-9da1-4f43-af70-02146f1ba405 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:17.506465 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f118d802-9da1-4f43-af70-02146f1ba405 service swift] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:17.519840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1159/2326] 10.222.0.26 () {60 vars in 1384 bytes} [Tue Sep 27 04:22:17 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 04:22:26.019366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24233f46-3d7e-4246-86e0-ba4204228528 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:26.020982 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-24233f46-3d7e-4246-86e0-ba4204228528 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:26.085542 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24233f46-3d7e-4246-86e0-ba4204228528 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:26.089268 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a84407-8b51-4e4f-a3d1-50446f09ca64 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:26.089964 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a84407-8b51-4e4f-a3d1-50446f09ca64 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:26.090523 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a84407-8b51-4e4f-a3d1-50446f09ca64 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:26.090523 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a84407-8b51-4e4f-a3d1-50446f09ca64 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:26.092367 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c5a84407-8b51-4e4f-a3d1-50446f09ca64 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:26.181781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a84407-8b51-4e4f-a3d1-50446f09ca64 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:26.183736 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a84407-8b51-4e4f-a3d1-50446f09ca64 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:26.197303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1168/2327] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:22:26 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:22:27.516518 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:DELETE=4,keystone:SELECT=39,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:22:32.945326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f118d802-9da1-4f43-af70-02146f1ba405 service swift] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:32.946832 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-f118d802-9da1-4f43-af70-02146f1ba405 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:33.016109 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f118d802-9da1-4f43-af70-02146f1ba405 service swift] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:33.021053 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7286cfcf-15bb-423c-8032-f70c8b574d35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:33.021981 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7286cfcf-15bb-423c-8032-f70c8b574d35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:33.023050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7286cfcf-15bb-423c-8032-f70c8b574d35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:33.023477 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7286cfcf-15bb-423c-8032-f70c8b574d35 admin admin] PATH_INFO: `/v3/users/7c13d3c029be4a56a6395521bc6b4bb7` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:33.024785 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7286cfcf-15bb-423c-8032-f70c8b574d35 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7c13d3c029be4a56a6395521bc6b4bb7)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:33.027122 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7286cfcf-15bb-423c-8032-f70c8b574d35 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:33.091001 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-7286cfcf-15bb-423c-8032-f70c8b574d35 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7c13d3c029be4a56a6395521bc6b4bb7'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:33.102263 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-7286cfcf-15bb-423c-8032-f70c8b574d35 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7c13d3c029be4a56a6395521bc6b4bb7'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:33.102263 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-7286cfcf-15bb-423c-8032-f70c8b574d35 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7c13d3c029be4a56a6395521bc6b4bb7'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:33.112127 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-7286cfcf-15bb-423c-8032-f70c8b574d35 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7c13d3c029be4a56a6395521bc6b4bb7'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:33.160254 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-7286cfcf-15bb-423c-8032-f70c8b574d35 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7c13d3c029be4a56a6395521bc6b4bb7'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:33.172996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1160/2328] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:22:32 2022] DELETE /identity/v3/users/7c13d3c029be4a56a6395521bc6b4bb7 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:22:33.179305 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bd9e43-ac3b-4a81-898c-99c89494daf8 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:33.180066 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bd9e43-ac3b-4a81-898c-99c89494daf8 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:33.180508 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bd9e43-ac3b-4a81-898c-99c89494daf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:33.216236 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-08bd9e43-ac3b-4a81-898c-99c89494daf8 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:22:33.217731 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-08bd9e43-ac3b-4a81-898c-99c89494daf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:33.270134 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:22:33.285651 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1169/2329] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:22:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:22:33.297138 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7286cfcf-15bb-423c-8032-f70c8b574d35 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:33.299436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-7286cfcf-15bb-423c-8032-f70c8b574d35 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:33.363765 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7286cfcf-15bb-423c-8032-f70c8b574d35 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:33.367904 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ae8e15-5883-4cb1-8b05-af136beb71f3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:33.368506 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ae8e15-5883-4cb1-8b05-af136beb71f3 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:33.368800 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ae8e15-5883-4cb1-8b05-af136beb71f3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:33.369118 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ae8e15-5883-4cb1-8b05-af136beb71f3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:33.379726 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ae8e15-5883-4cb1-8b05-af136beb71f3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:33.381151 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ae8e15-5883-4cb1-8b05-af136beb71f3 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:33.395979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1161/2330] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:22:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:22:33.698372 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bd9e43-ac3b-4a81-898c-99c89494daf8 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:33.702794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-08bd9e43-ac3b-4a81-898c-99c89494daf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:33.721157 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bd9e43-ac3b-4a81-898c-99c89494daf8 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:33.734512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606d01fb-ff4d-4685-be5e-0557b3840d16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:33.735155 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606d01fb-ff4d-4685-be5e-0557b3840d16 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:33.735431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606d01fb-ff4d-4685-be5e-0557b3840d16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:33.735709 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606d01fb-ff4d-4685-be5e-0557b3840d16 admin admin] PATH_INFO: `/v3/projects/a613cbeb387243ccb4d167d82c5068de` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:33.736847 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606d01fb-ff4d-4685-be5e-0557b3840d16 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a613cbeb387243ccb4d167d82c5068de)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:33.738828 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606d01fb-ff4d-4685-be5e-0557b3840d16 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:33.752920 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c02bcbb-2699-4239-8a77-2539bcadee6d None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:33.753329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c02bcbb-2699-4239-8a77-2539bcadee6d None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:33.753603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c02bcbb-2699-4239-8a77-2539bcadee6d None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:33.755242 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1162/2331] 10.222.0.26 () {54 vars in 921 bytes} [Tue Sep 27 04:22:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:22:33.761599 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40368b16-5f31-45ec-afed-d66e44943ed4 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:33.761970 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40368b16-5f31-45ec-afed-d66e44943ed4 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:33.762171 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40368b16-5f31-45ec-afed-d66e44943ed4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:33.784561 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-40368b16-5f31-45ec-afed-d66e44943ed4 None None] MFA Rules not processed for user `a1586ee841cd44658e4d5feb25de7940`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:22:33.785691 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-40368b16-5f31-45ec-afed-d66e44943ed4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:33.806850 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-606d01fb-ff4d-4685-be5e-0557b3840d16 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a613cbeb387243ccb4d167d82c5068de', 'request_id': 'req-606d01fb-ff4d-4685-be5e-0557b3840d16'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:33.816419 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-606d01fb-ff4d-4685-be5e-0557b3840d16 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a613cbeb387243ccb4d167d82c5068de', 'request_id': 'req-606d01fb-ff4d-4685-be5e-0557b3840d16'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:33.824544 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-606d01fb-ff4d-4685-be5e-0557b3840d16 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a613cbeb387243ccb4d167d82c5068de', 'request_id': 'req-606d01fb-ff4d-4685-be5e-0557b3840d16'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:33.830931 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:22:33.842058 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-606d01fb-ff4d-4685-be5e-0557b3840d16 admin admin] The token cache is being invalidate because project a613cbeb387243ccb4d167d82c5068de was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:22:33.842445 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-606d01fb-ff4d-4685-be5e-0557b3840d16 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:33.843695 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-606d01fb-ff4d-4685-be5e-0557b3840d16 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:33.845765 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1170/2332] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:22:33 2022] DELETE /identity/v3/projects/a613cbeb387243ccb4d167d82c5068de => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:22:33.851209 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b81cb1-c117-4af9-8280-35dbae42358d None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:33.851448 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b81cb1-c117-4af9-8280-35dbae42358d None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:33.851870 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b81cb1-c117-4af9-8280-35dbae42358d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:33.888630 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-32b81cb1-c117-4af9-8280-35dbae42358d None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:22:33.891081 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-32b81cb1-c117-4af9-8280-35dbae42358d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:33.928442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1163/2333] 10.222.0.26 () {58 vars in 1018 bytes} [Tue Sep 27 04:22:33 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:22:33.945562 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40368b16-5f31-45ec-afed-d66e44943ed4 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:33.950458 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-40368b16-5f31-45ec-afed-d66e44943ed4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:34.009785 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1171/2334] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:22:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:22:34.018620 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464ce18c-07f7-45fb-866b-e52fdc54bfd9 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:34.018902 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464ce18c-07f7-45fb-866b-e52fdc54bfd9 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:34.019176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464ce18c-07f7-45fb-866b-e52fdc54bfd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:34.027027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40368b16-5f31-45ec-afed-d66e44943ed4 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:34.046497 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-464ce18c-07f7-45fb-866b-e52fdc54bfd9 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:22:34.047750 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-464ce18c-07f7-45fb-866b-e52fdc54bfd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:34.057575 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1172/2335] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:22:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:22:34.062117 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464ce18c-07f7-45fb-866b-e52fdc54bfd9 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:34.079223 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464ce18c-07f7-45fb-866b-e52fdc54bfd9 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:34.085521 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2e2fbc-7d96-48c7-b995-14a2b0717908 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:34.086235 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2e2fbc-7d96-48c7-b995-14a2b0717908 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:34.086515 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2e2fbc-7d96-48c7-b995-14a2b0717908 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:34.086789 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2e2fbc-7d96-48c7-b995-14a2b0717908 admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:34.087506 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2e2fbc-7d96-48c7-b995-14a2b0717908 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:22:34.087811 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2e2fbc-7d96-48c7-b995-14a2b0717908 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:34.089912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2e2fbc-7d96-48c7-b995-14a2b0717908 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:34.108104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa2c6f0-7b39-4151-ba00-d4dcbffc788b service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dcc53332e6b462baf349a72b19d2b3b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:34.108104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa2c6f0-7b39-4151-ba00-d4dcbffc788b service glance] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:34.108104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa2c6f0-7b39-4151-ba00-d4dcbffc788b service glance] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:34.108104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa2c6f0-7b39-4151-ba00-d4dcbffc788b service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:34.109681 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1173/2336] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:22:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:22:34.110590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2e2fbc-7d96-48c7-b995-14a2b0717908 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:34.122060 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2e2fbc-7d96-48c7-b995-14a2b0717908 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:34.127745 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9263b932-e46d-4d69-bcd0-01d23319cd07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:34.131687 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9263b932-e46d-4d69-bcd0-01d23319cd07 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:34.131687 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9263b932-e46d-4d69-bcd0-01d23319cd07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:34.131687 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9263b932-e46d-4d69-bcd0-01d23319cd07 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:34.131687 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9263b932-e46d-4d69-bcd0-01d23319cd07 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:34.132959 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9263b932-e46d-4d69-bcd0-01d23319cd07 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:34.173779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1174/2337] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:22:34 2022] POST /identity/v3/projects => generated 391 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:22:34.175143 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9263b932-e46d-4d69-bcd0-01d23319cd07 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:34.188935 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa2c6f0-7b39-4151-ba00-d4dcbffc788b service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:34.190404 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa2c6f0-7b39-4151-ba00-d4dcbffc788b service glance] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:34.203391 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1164/2338] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:22:33 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 258 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:22:34.219298 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:22:34.242225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9263b932-e46d-4d69-bcd0-01d23319cd07 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:34.246851 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67f1725-938d-4a07-a85f-6489af6ded54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:34.247449 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67f1725-938d-4a07-a85f-6489af6ded54 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:34.247844 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67f1725-938d-4a07-a85f-6489af6ded54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:34.248120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67f1725-938d-4a07-a85f-6489af6ded54 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:34.248745 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67f1725-938d-4a07-a85f-6489af6ded54 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:34.250711 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67f1725-938d-4a07-a85f-6489af6ded54 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:34.272736 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1175/2339] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:22:34 2022] POST /identity/v3/users => generated 296 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:22:34.276842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67f1725-938d-4a07-a85f-6489af6ded54 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:34.277054 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9901ad-b3be-4a3a-b199-558f4241b2bf None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:34.277391 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9901ad-b3be-4a3a-b199-558f4241b2bf None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:34.277673 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9901ad-b3be-4a3a-b199-558f4241b2bf None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:34.279062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1165/2340] 10.222.0.26 () {54 vars in 921 bytes} [Tue Sep 27 04:22:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:22:34.285135 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a0bd72-5d2e-4887-bbf4-03268b28f752 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:34.285473 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a0bd72-5d2e-4887-bbf4-03268b28f752 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:34.285697 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a0bd72-5d2e-4887-bbf4-03268b28f752 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:34.288003 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67f1725-938d-4a07-a85f-6489af6ded54 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:34.292511 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7f044c-ce6b-4309-9a78-ea6c11fd9aa6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:34.293099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7f044c-ce6b-4309-9a78-ea6c11fd9aa6 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:34.293415 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7f044c-ce6b-4309-9a78-ea6c11fd9aa6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:34.293683 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7f044c-ce6b-4309-9a78-ea6c11fd9aa6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:34.294345 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7f044c-ce6b-4309-9a78-ea6c11fd9aa6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:34.295941 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7f044c-ce6b-4309-9a78-ea6c11fd9aa6 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:34.305433 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-c7a0bd72-5d2e-4887-bbf4-03268b28f752 None None] MFA Rules not processed for user `cf29a8da103b411fbc79004eb371c05d`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:22:34.305603 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1176/2341] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:22:34 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:22:34.306671 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c7a0bd72-5d2e-4887-bbf4-03268b28f752 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:34.309543 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7f044c-ce6b-4309-9a78-ea6c11fd9aa6 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:34.312726 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:22:34.319673 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7f044c-ce6b-4309-9a78-ea6c11fd9aa6 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:34.324144 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b8c8c2-633b-4b19-8cee-7aef3af14580 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:34.324716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b8c8c2-633b-4b19-8cee-7aef3af14580 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:34.324989 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b8c8c2-633b-4b19-8cee-7aef3af14580 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:34.325259 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b8c8c2-633b-4b19-8cee-7aef3af14580 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:34.325901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b8c8c2-633b-4b19-8cee-7aef3af14580 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:34.328086 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b8c8c2-633b-4b19-8cee-7aef3af14580 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:34.341490 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1177/2342] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:22:34 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:22:34.345850 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b8c8c2-633b-4b19-8cee-7aef3af14580 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:34.355667 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b8c8c2-633b-4b19-8cee-7aef3af14580 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:34.360289 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defe280c-c604-405c-9717-2123b8cb87bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:34.360925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defe280c-c604-405c-9717-2123b8cb87bd admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:34.361218 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defe280c-c604-405c-9717-2123b8cb87bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:34.361530 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defe280c-c604-405c-9717-2123b8cb87bd admin admin] PATH_INFO: `/v3/projects/ff9b9972c6b646659b6defa8ffcde374/users/c386a6e1db024e03abf012e16b2d40ae/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:34.364666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1166/2343] 10.222.0.26 () {58 vars in 1018 bytes} [Tue Sep 27 04:22:34 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:22:34.376740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defe280c-c604-405c-9717-2123b8cb87bd admin admin] RBAC: Authorizing `identity:create_grant(project_id=ff9b9972c6b646659b6defa8ffcde374, user_id=c386a6e1db024e03abf012e16b2d40ae, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:34.377337 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a0bd72-5d2e-4887-bbf4-03268b28f752 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:34.378684 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c7a0bd72-5d2e-4887-bbf4-03268b28f752 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:34.379396 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defe280c-c604-405c-9717-2123b8cb87bd admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:34.390201 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a0bd72-5d2e-4887-bbf4-03268b28f752 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:34.394571 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1178/2344] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:22:34 2022] PUT /identity/v3/projects/ff9b9972c6b646659b6defa8ffcde374/users/c386a6e1db024e03abf012e16b2d40ae/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:22:34.404636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3c95b7-c776-4a73-be4b-854b1dff2892 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:34.404923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3c95b7-c776-4a73-be4b-854b1dff2892 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:34.405286 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3c95b7-c776-4a73-be4b-854b1dff2892 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:34.444850 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-6f3c95b7-c776-4a73-be4b-854b1dff2892 None None] MFA Rules not processed for user `c386a6e1db024e03abf012e16b2d40ae`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:22:34.446553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6f3c95b7-c776-4a73-be4b-854b1dff2892 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:34.451349 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b66a77-4cd8-4cfb-b32f-eb84e6bd3b65 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:34.452006 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b66a77-4cd8-4cfb-b32f-eb84e6bd3b65 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:34.452925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b66a77-4cd8-4cfb-b32f-eb84e6bd3b65 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:34.453206 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b66a77-4cd8-4cfb-b32f-eb84e6bd3b65 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:34.461027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b66a77-4cd8-4cfb-b32f-eb84e6bd3b65 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:34.462505 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b66a77-4cd8-4cfb-b32f-eb84e6bd3b65 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:34.472262 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1167/2345] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:22:34 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:22:34.524659 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1179/2346] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:22:34 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:22:34.537499 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b66a77-4cd8-4cfb-b32f-eb84e6bd3b65 service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:34.539093 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-36b66a77-4cd8-4cfb-b32f-eb84e6bd3b65 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:34.562790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d445b87e-ac96-4692-ad34-1710f65759d8 None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:34.563064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d445b87e-ac96-4692-ad34-1710f65759d8 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:34.563335 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d445b87e-ac96-4692-ad34-1710f65759d8 None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:34.564790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1180/2347] 10.222.0.26 () {54 vars in 921 bytes} [Tue Sep 27 04:22:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 04:22:34.571096 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12653299-c0f7-4c46-b140-d26d8855e27b None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:34.571455 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12653299-c0f7-4c46-b140-d26d8855e27b None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:34.571787 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12653299-c0f7-4c46-b140-d26d8855e27b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:34.593677 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-12653299-c0f7-4c46-b140-d26d8855e27b None None] MFA Rules not processed for user `a1586ee841cd44658e4d5feb25de7940`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:22:34.595030 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-12653299-c0f7-4c46-b140-d26d8855e27b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:34.602642 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b66a77-4cd8-4cfb-b32f-eb84e6bd3b65 service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:34.606361 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e8978e-d613-4f9a-89ee-7b6217fd3fc5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:34.607037 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e8978e-d613-4f9a-89ee-7b6217fd3fc5 service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:34.607403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e8978e-d613-4f9a-89ee-7b6217fd3fc5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:34.607743 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e8978e-d613-4f9a-89ee-7b6217fd3fc5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:34.616116 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e8978e-d613-4f9a-89ee-7b6217fd3fc5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:34.617547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e8978e-d613-4f9a-89ee-7b6217fd3fc5 service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:34.626828 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1168/2348] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:22:34 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:22:34.666018 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1181/2349] 10.222.0.26 () {58 vars in 1018 bytes} [Tue Sep 27 04:22:34 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:22:34.677325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e8978e-d613-4f9a-89ee-7b6217fd3fc5 service cinder] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:34.687640 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e8978e-d613-4f9a-89ee-7b6217fd3fc5 service cinder] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:34.691274 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e192e10-afc0-43a9-b22f-c5ff5b061418 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:34.691860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e192e10-afc0-43a9-b22f-c5ff5b061418 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:34.692139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e192e10-afc0-43a9-b22f-c5ff5b061418 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:34.692411 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e192e10-afc0-43a9-b22f-c5ff5b061418 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:34.702931 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e192e10-afc0-43a9-b22f-c5ff5b061418 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:34.705134 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e192e10-afc0-43a9-b22f-c5ff5b061418 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:34.723148 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1169/2350] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:22:34 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:22:40.352370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87392215-9892-4aab-82e6-cf27826a36ce None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:40.352788 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87392215-9892-4aab-82e6-cf27826a36ce None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:40.353043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87392215-9892-4aab-82e6-cf27826a36ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:40.360137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:22:40.374474 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-87392215-9892-4aab-82e6-cf27826a36ce None None] MFA Rules not processed for user `5a2fcc8c9ac44dceadcc30d163bbfa0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:22:40.375656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-87392215-9892-4aab-82e6-cf27826a36ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:40.435223 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1182/2351] 10.222.0.26 () {58 vars in 1009 bytes} [Tue Sep 27 04:22:40 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:22:40.448015 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e192e10-afc0-43a9-b22f-c5ff5b061418 service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:40.449417 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-7e192e10-afc0-43a9-b22f-c5ff5b061418 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:40.508788 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:22:40.532945 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e192e10-afc0-43a9-b22f-c5ff5b061418 service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:40.538087 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cc959d-db4a-4dda-97e0-a36ed23bd8b0 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '181dd5dc720c4a97860852062b7b19c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:40.538701 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cc959d-db4a-4dda-97e0-a36ed23bd8b0 service swift] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:40.538974 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cc959d-db4a-4dda-97e0-a36ed23bd8b0 service swift] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:40.539282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cc959d-db4a-4dda-97e0-a36ed23bd8b0 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:40.548352 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23cc959d-db4a-4dda-97e0-a36ed23bd8b0 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:40.549959 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23cc959d-db4a-4dda-97e0-a36ed23bd8b0 service swift] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:40.558474 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1170/2352] 10.222.0.26 () {60 vars in 1384 bytes} [Tue Sep 27 04:22:40 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 04:22:50.428550 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=24 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:22:50.555102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=8 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:22:56.246905 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87392215-9892-4aab-82e6-cf27826a36ce None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:56.307118 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87392215-9892-4aab-82e6-cf27826a36ce None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:56.311485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b58e26-721a-44d7-b47e-436e3bd96a4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:56.312119 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b58e26-721a-44d7-b47e-436e3bd96a4f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:56.312392 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b58e26-721a-44d7-b47e-436e3bd96a4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:56.312657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b58e26-721a-44d7-b47e-436e3bd96a4f admin admin] PATH_INFO: `/v3/users/c386a6e1db024e03abf012e16b2d40ae` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:56.313815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b58e26-721a-44d7-b47e-436e3bd96a4f admin admin] RBAC: Authorizing `identity:delete_user(user_id=c386a6e1db024e03abf012e16b2d40ae)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:56.315692 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b58e26-721a-44d7-b47e-436e3bd96a4f admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:56.363312 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-63b58e26-721a-44d7-b47e-436e3bd96a4f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c386a6e1db024e03abf012e16b2d40ae'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:56.370098 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-63b58e26-721a-44d7-b47e-436e3bd96a4f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c386a6e1db024e03abf012e16b2d40ae'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:56.370773 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-63b58e26-721a-44d7-b47e-436e3bd96a4f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c386a6e1db024e03abf012e16b2d40ae'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:56.377733 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-63b58e26-721a-44d7-b47e-436e3bd96a4f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c386a6e1db024e03abf012e16b2d40ae'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:56.389505 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-63b58e26-721a-44d7-b47e-436e3bd96a4f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c386a6e1db024e03abf012e16b2d40ae'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:56.395796 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1183/2353] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:22:56 2022] DELETE /identity/v3/users/c386a6e1db024e03abf012e16b2d40ae => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:22:56.400209 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f00946d-0da5-4ecc-aa8a-374d812aad7d None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:56.400779 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f00946d-0da5-4ecc-aa8a-374d812aad7d None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:56.400779 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f00946d-0da5-4ecc-aa8a-374d812aad7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:56.420320 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-1f00946d-0da5-4ecc-aa8a-374d812aad7d None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:22:56.421534 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-1f00946d-0da5-4ecc-aa8a-374d812aad7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:56.478757 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1171/2354] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:22:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:22:56.489081 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b58e26-721a-44d7-b47e-436e3bd96a4f admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:56.490488 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-63b58e26-721a-44d7-b47e-436e3bd96a4f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:56.545924 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b58e26-721a-44d7-b47e-436e3bd96a4f admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:56.550101 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0488f0e0-94a9-4732-b395-542761b2c9a2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:56.550699 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0488f0e0-94a9-4732-b395-542761b2c9a2 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:56.550976 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0488f0e0-94a9-4732-b395-542761b2c9a2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:56.551246 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0488f0e0-94a9-4732-b395-542761b2c9a2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:56.559837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0488f0e0-94a9-4732-b395-542761b2c9a2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:56.561247 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0488f0e0-94a9-4732-b395-542761b2c9a2 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:56.571034 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1184/2355] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:22:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:22:56.842066 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f00946d-0da5-4ecc-aa8a-374d812aad7d None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:22:56.843691 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-1f00946d-0da5-4ecc-aa8a-374d812aad7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:22:56.857555 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f00946d-0da5-4ecc-aa8a-374d812aad7d None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:22:56.871337 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48f9e08-f7c7-4f1d-8878-191db451a589 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:22:56.871923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48f9e08-f7c7-4f1d-8878-191db451a589 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:22:56.872206 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48f9e08-f7c7-4f1d-8878-191db451a589 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:22:56.872482 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48f9e08-f7c7-4f1d-8878-191db451a589 admin admin] PATH_INFO: `/v3/projects/ff9b9972c6b646659b6defa8ffcde374` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:22:56.875754 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48f9e08-f7c7-4f1d-8878-191db451a589 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ff9b9972c6b646659b6defa8ffcde374)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:22:56.875754 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48f9e08-f7c7-4f1d-8878-191db451a589 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:22:56.927995 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-c48f9e08-f7c7-4f1d-8878-191db451a589 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ff9b9972c6b646659b6defa8ffcde374', 'request_id': 'req-c48f9e08-f7c7-4f1d-8878-191db451a589'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:56.934163 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-c48f9e08-f7c7-4f1d-8878-191db451a589 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ff9b9972c6b646659b6defa8ffcde374', 'request_id': 'req-c48f9e08-f7c7-4f1d-8878-191db451a589'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:56.940908 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-c48f9e08-f7c7-4f1d-8878-191db451a589 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ff9b9972c6b646659b6defa8ffcde374', 'request_id': 'req-c48f9e08-f7c7-4f1d-8878-191db451a589'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:56.956112 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-c48f9e08-f7c7-4f1d-8878-191db451a589 admin admin] The token cache is being invalidate because project ff9b9972c6b646659b6defa8ffcde374 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:22:56.956400 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-c48f9e08-f7c7-4f1d-8878-191db451a589 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:56.957208 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-c48f9e08-f7c7-4f1d-8878-191db451a589 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:22:56.965012 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1172/2356] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:22:56 2022] DELETE /identity/v3/projects/ff9b9972c6b646659b6defa8ffcde374 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:23:04.615418 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca1398f-1c7b-4b18-bcf9-95e50b6374b4 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:23:04.615894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca1398f-1c7b-4b18-bcf9-95e50b6374b4 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:23:04.616037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca1398f-1c7b-4b18-bcf9-95e50b6374b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:23:04.634974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-eca1398f-1c7b-4b18-bcf9-95e50b6374b4 None None] MFA Rules not processed for user `2d0ba2ae3ffe4bb9b99f1347719df411`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:23:04.636147 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-eca1398f-1c7b-4b18-bcf9-95e50b6374b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:23:04.655323 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:23:04.699470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1185/2357] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:23:04 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:23:06.952126 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=54,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:23:07.461872 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48f9e08-f7c7-4f1d-8878-191db451a589 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:23:07.463298 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c48f9e08-f7c7-4f1d-8878-191db451a589 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:23:07.581888 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48f9e08-f7c7-4f1d-8878-191db451a589 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:23:07.587915 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34a889f-a3d2-4595-8c63-477d50e6919f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3a1bc198dfc0486e9e56d4fb28cda05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:23:07.588288 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34a889f-a3d2-4595-8c63-477d50e6919f service placement] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:23:07.588566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34a889f-a3d2-4595-8c63-477d50e6919f service placement] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:23:07.588840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34a889f-a3d2-4595-8c63-477d50e6919f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:23:07.590819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-d34a889f-a3d2-4595-8c63-477d50e6919f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:23:07.617078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d34a889f-a3d2-4595-8c63-477d50e6919f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:23:07.618728 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d34a889f-a3d2-4595-8c63-477d50e6919f service placement] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:23:07.629537 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1173/2358] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:23:07 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:23:14.692691 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=15 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:23:17.041886 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f14c151-23bf-47b1-b640-92539d860d93 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:23:17.042437 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f14c151-23bf-47b1-b640-92539d860d93 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:23:17.042437 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f14c151-23bf-47b1-b640-92539d860d93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:23:17.069532 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-7f14c151-23bf-47b1-b640-92539d860d93 None None] MFA Rules not processed for user `5a2fcc8c9ac44dceadcc30d163bbfa0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:23:17.071436 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-7f14c151-23bf-47b1-b640-92539d860d93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:23:17.142473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1186/2359] 10.222.0.26 () {58 vars in 967 bytes} [Tue Sep 27 04:23:17 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:23:17.156335 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34a889f-a3d2-4595-8c63-477d50e6919f service placement] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:23:17.159093 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-d34a889f-a3d2-4595-8c63-477d50e6919f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:23:17.260145 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34a889f-a3d2-4595-8c63-477d50e6919f service placement] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:23:17.270911 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209974ca-79a4-4dbd-9570-458d1c680dcf service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '181dd5dc720c4a97860852062b7b19c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:23:17.271851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209974ca-79a4-4dbd-9570-458d1c680dcf service swift] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:23:17.274446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209974ca-79a4-4dbd-9570-458d1c680dcf service swift] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:23:17.274947 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209974ca-79a4-4dbd-9570-458d1c680dcf service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:23:17.300354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-209974ca-79a4-4dbd-9570-458d1c680dcf service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:23:17.302162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-209974ca-79a4-4dbd-9570-458d1c680dcf service swift] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:23:17.311367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1174/2360] 10.222.0.26 () {60 vars in 1384 bytes} [Tue Sep 27 04:23:17 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 04:23:27.133893 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=25 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:23:27.307643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=64 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:23:36.727885 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35af7740-a252-4586-8c3f-c34b646a9836 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:23:36.728352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35af7740-a252-4586-8c3f-c34b646a9836 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:23:36.728450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35af7740-a252-4586-8c3f-c34b646a9836 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:23:36.747319 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-35af7740-a252-4586-8c3f-c34b646a9836 None None] MFA Rules not processed for user `782ebdc88e21483eb3cecd00415b9b07`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:23:36.748503 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-35af7740-a252-4586-8c3f-c34b646a9836 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:23:36.804208 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1187/2361] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:23:36 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:23:37.033628 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488c1fb9-c2c6-4f71-91db-7ff39ac9486b None None] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:23:37.034144 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488c1fb9-c2c6-4f71-91db-7ff39ac9486b None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:23:37.034269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488c1fb9-c2c6-4f71-91db-7ff39ac9486b None None] PATH_INFO: `/` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:23:37.035557 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1175/2362] 10.222.0.26 () {54 vars in 912 bytes} [Tue Sep 27 04:23:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:23:37.040417 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc96597-d130-4b88-b5eb-9a2932c3fb38 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:23:37.040699 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc96597-d130-4b88-b5eb-9a2932c3fb38 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:23:37.040978 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc96597-d130-4b88-b5eb-9a2932c3fb38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:23:37.061454 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-ddc96597-d130-4b88-b5eb-9a2932c3fb38 None None] MFA Rules not processed for user `cf29a8da103b411fbc79004eb371c05d`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:23:37.062671 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ddc96597-d130-4b88-b5eb-9a2932c3fb38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:23:37.114710 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1188/2363] 10.222.0.26 () {58 vars in 1009 bytes} [Tue Sep 27 04:23:37 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:23:37.126124 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488c1fb9-c2c6-4f71-91db-7ff39ac9486b None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:23:37.127829 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-488c1fb9-c2c6-4f71-91db-7ff39ac9486b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:23:37.141215 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488c1fb9-c2c6-4f71-91db-7ff39ac9486b None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:23:37.144597 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d179b650-52a8-4e13-a88a-40c4a2f729d5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:23:37.145177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d179b650-52a8-4e13-a88a-40c4a2f729d5 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:23:37.145452 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d179b650-52a8-4e13-a88a-40c4a2f729d5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:23:37.145720 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d179b650-52a8-4e13-a88a-40c4a2f729d5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:23:37.152479 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d179b650-52a8-4e13-a88a-40c4a2f729d5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:23:37.153889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d179b650-52a8-4e13-a88a-40c4a2f729d5 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:23:37.162121 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1176/2364] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:23:37 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:23:47.108582 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=48 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:23:47.158078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=6 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:24:28.279604 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66eec08c-343b-474e-995c-96d6d7ea7e72 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:28.280121 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66eec08c-343b-474e-995c-96d6d7ea7e72 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:28.280227 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66eec08c-343b-474e-995c-96d6d7ea7e72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:28.299429 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-66eec08c-343b-474e-995c-96d6d7ea7e72 None None] MFA Rules not processed for user `7c56e7333ce4450b8cbd5caeafc73947`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:24:28.300687 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-66eec08c-343b-474e-995c-96d6d7ea7e72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:24:28.358887 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1189/2365] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:24:28 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:24:35.695676 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d179b650-52a8-4e13-a88a-40c4a2f729d5 service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:24:35.697134 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-d179b650-52a8-4e13-a88a-40c4a2f729d5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:24:35.766348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d179b650-52a8-4e13-a88a-40c4a2f729d5 service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:24:35.770799 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f53991-99aa-4652-afd9-4466677a6074 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:24:35.771413 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f53991-99aa-4652-afd9-4466677a6074 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:35.771690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f53991-99aa-4652-afd9-4466677a6074 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:35.772603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f53991-99aa-4652-afd9-4466677a6074 admin admin] PATH_INFO: `/v3/users/782ebdc88e21483eb3cecd00415b9b07` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:35.773761 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f53991-99aa-4652-afd9-4466677a6074 admin admin] RBAC: Authorizing `identity:delete_user(user_id=782ebdc88e21483eb3cecd00415b9b07)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:24:35.775647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f53991-99aa-4652-afd9-4466677a6074 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:24:35.822144 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-33f53991-99aa-4652-afd9-4466677a6074 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '782ebdc88e21483eb3cecd00415b9b07'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:35.828781 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-33f53991-99aa-4652-afd9-4466677a6074 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '782ebdc88e21483eb3cecd00415b9b07'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:35.829441 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-33f53991-99aa-4652-afd9-4466677a6074 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '782ebdc88e21483eb3cecd00415b9b07'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:35.837260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-33f53991-99aa-4652-afd9-4466677a6074 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '782ebdc88e21483eb3cecd00415b9b07'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:35.849106 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-33f53991-99aa-4652-afd9-4466677a6074 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '782ebdc88e21483eb3cecd00415b9b07'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:35.854975 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1177/2366] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:24:35 2022] DELETE /identity/v3/users/782ebdc88e21483eb3cecd00415b9b07 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:24:36.035509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66eec08c-343b-474e-995c-96d6d7ea7e72 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:24:36.050097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66eec08c-343b-474e-995c-96d6d7ea7e72 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:24:36.127653 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e303b38e-bde6-420f-ac55-6e79879db24c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:24:36.128425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e303b38e-bde6-420f-ac55-6e79879db24c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:36.128796 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e303b38e-bde6-420f-ac55-6e79879db24c admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:36.129136 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e303b38e-bde6-420f-ac55-6e79879db24c admin admin] PATH_INFO: `/v3/projects/14d42cf78c32468cb8bf10aa847c7cc3` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:36.132579 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e303b38e-bde6-420f-ac55-6e79879db24c admin admin] RBAC: Authorizing `identity:delete_project(project_id=14d42cf78c32468cb8bf10aa847c7cc3)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:24:36.134687 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e303b38e-bde6-420f-ac55-6e79879db24c admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:24:36.195253 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-e303b38e-bde6-420f-ac55-6e79879db24c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '14d42cf78c32468cb8bf10aa847c7cc3', 'request_id': 'req-e303b38e-bde6-420f-ac55-6e79879db24c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:36.202586 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-e303b38e-bde6-420f-ac55-6e79879db24c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '14d42cf78c32468cb8bf10aa847c7cc3', 'request_id': 'req-e303b38e-bde6-420f-ac55-6e79879db24c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:36.209831 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-e303b38e-bde6-420f-ac55-6e79879db24c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '14d42cf78c32468cb8bf10aa847c7cc3', 'request_id': 'req-e303b38e-bde6-420f-ac55-6e79879db24c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:36.226271 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-e303b38e-bde6-420f-ac55-6e79879db24c admin admin] The token cache is being invalidate because project 14d42cf78c32468cb8bf10aa847c7cc3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:24:36.226560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-e303b38e-bde6-420f-ac55-6e79879db24c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:36.227189 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-e303b38e-bde6-420f-ac55-6e79879db24c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:36.232753 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1190/2367] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:24:36 2022] DELETE /identity/v3/projects/14d42cf78c32468cb8bf10aa847c7cc3 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:24:36.234232 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f6dc5b-de41-428e-89f5-27b34e34dadf None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:36.234514 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f6dc5b-de41-428e-89f5-27b34e34dadf None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:36.234787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f6dc5b-de41-428e-89f5-27b34e34dadf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:36.257663 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-19f6dc5b-de41-428e-89f5-27b34e34dadf None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:24:36.258916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-19f6dc5b-de41-428e-89f5-27b34e34dadf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:24:36.325514 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1178/2368] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:24:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:24:36.333319 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edf8e27-be2e-44ec-8e1f-63db827f6728 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:36.333609 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edf8e27-be2e-44ec-8e1f-63db827f6728 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:36.333936 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edf8e27-be2e-44ec-8e1f-63db827f6728 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:36.354099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-7edf8e27-be2e-44ec-8e1f-63db827f6728 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:24:36.355300 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-7edf8e27-be2e-44ec-8e1f-63db827f6728 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:24:36.363154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1191/2369] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:24:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:24:36.368243 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f6dc5b-de41-428e-89f5-27b34e34dadf None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:24:36.378001 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f6dc5b-de41-428e-89f5-27b34e34dadf None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:24:36.382616 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2e13da-11be-44d3-bc2b-1b3e9ed462c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:24:36.383213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2e13da-11be-44d3-bc2b-1b3e9ed462c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:36.383490 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2e13da-11be-44d3-bc2b-1b3e9ed462c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:36.383761 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2e13da-11be-44d3-bc2b-1b3e9ed462c2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:36.384431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2e13da-11be-44d3-bc2b-1b3e9ed462c2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:24:36.384909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2e13da-11be-44d3-bc2b-1b3e9ed462c2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:24:36.386535 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2e13da-11be-44d3-bc2b-1b3e9ed462c2 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:24:36.398318 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1179/2370] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:24:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:24:36.402548 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edf8e27-be2e-44ec-8e1f-63db827f6728 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:24:36.412311 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edf8e27-be2e-44ec-8e1f-63db827f6728 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:24:36.423985 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893635d9-2c51-47b8-9272-a99b321efb07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:24:36.424595 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893635d9-2c51-47b8-9272-a99b321efb07 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:36.424867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893635d9-2c51-47b8-9272-a99b321efb07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:36.425136 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893635d9-2c51-47b8-9272-a99b321efb07 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:36.425853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893635d9-2c51-47b8-9272-a99b321efb07 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:24:36.427742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893635d9-2c51-47b8-9272-a99b321efb07 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:24:36.455864 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1192/2371] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:24:36 2022] POST /identity/v3/projects => generated 373 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:24:36.460210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2e13da-11be-44d3-bc2b-1b3e9ed462c2 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:24:36.470101 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2e13da-11be-44d3-bc2b-1b3e9ed462c2 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:24:36.473602 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:24:36.558759 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81084e6a-b576-4902-8759-642868453746 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:24:36.559674 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81084e6a-b576-4902-8759-642868453746 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:36.560152 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81084e6a-b576-4902-8759-642868453746 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:36.560856 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81084e6a-b576-4902-8759-642868453746 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:36.561099 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81084e6a-b576-4902-8759-642868453746 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:24:36.563437 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81084e6a-b576-4902-8759-642868453746 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:24:36.594357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1180/2372] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:24:36 2022] POST /identity/v3/users => generated 287 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:24:36.594736 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893635d9-2c51-47b8-9272-a99b321efb07 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:24:36.608025 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893635d9-2c51-47b8-9272-a99b321efb07 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:24:36.615461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4433ab03-f1fd-4dad-8314-e14bc1da1617 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:24:36.616816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4433ab03-f1fd-4dad-8314-e14bc1da1617 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:36.617099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4433ab03-f1fd-4dad-8314-e14bc1da1617 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:36.617372 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4433ab03-f1fd-4dad-8314-e14bc1da1617 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:36.618057 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4433ab03-f1fd-4dad-8314-e14bc1da1617 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:24:36.619656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4433ab03-f1fd-4dad-8314-e14bc1da1617 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:24:36.621277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:24:36.637885 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1193/2373] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:24:36 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:24:36.639197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81084e6a-b576-4902-8759-642868453746 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:24:36.649706 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81084e6a-b576-4902-8759-642868453746 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:24:36.655834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d2e65a-ccc4-4b38-9259-b423164e0d85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:24:36.656746 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d2e65a-ccc4-4b38-9259-b423164e0d85 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:36.657183 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d2e65a-ccc4-4b38-9259-b423164e0d85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:36.657626 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d2e65a-ccc4-4b38-9259-b423164e0d85 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:36.658792 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d2e65a-ccc4-4b38-9259-b423164e0d85 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:24:36.661146 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d2e65a-ccc4-4b38-9259-b423164e0d85 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:24:36.676701 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1181/2374] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:24:36 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:24:36.681484 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4433ab03-f1fd-4dad-8314-e14bc1da1617 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:24:36.694372 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4433ab03-f1fd-4dad-8314-e14bc1da1617 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:24:36.699650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fae26a4-22ae-4196-b30e-4c79b9f2bd03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:24:36.700528 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fae26a4-22ae-4196-b30e-4c79b9f2bd03 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:36.700962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fae26a4-22ae-4196-b30e-4c79b9f2bd03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:36.701435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fae26a4-22ae-4196-b30e-4c79b9f2bd03 admin admin] PATH_INFO: `/v3/projects/1e595829629a4dd4a7fccbdabc896cb0/users/5ebc33b6c372434287b632d70f0a9e83/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:36.718381 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fae26a4-22ae-4196-b30e-4c79b9f2bd03 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1e595829629a4dd4a7fccbdabc896cb0, user_id=5ebc33b6c372434287b632d70f0a9e83, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:24:36.721868 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fae26a4-22ae-4196-b30e-4c79b9f2bd03 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:24:36.739975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1194/2375] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:24:36 2022] PUT /identity/v3/projects/1e595829629a4dd4a7fccbdabc896cb0/users/5ebc33b6c372434287b632d70f0a9e83/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:24:36.746604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae926e8-918a-4157-8679-ce0de88e8fb0 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:36.747501 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae926e8-918a-4157-8679-ce0de88e8fb0 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:36.748273 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae926e8-918a-4157-8679-ce0de88e8fb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:36.771040 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-5ae926e8-918a-4157-8679-ce0de88e8fb0 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:24:36.772619 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-5ae926e8-918a-4157-8679-ce0de88e8fb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:24:36.850915 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1182/2376] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:24:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:24:36.857646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fae26a4-22ae-4196-b30e-4c79b9f2bd03 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:24:36.859958 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-0fae26a4-22ae-4196-b30e-4c79b9f2bd03 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:24:36.944728 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fae26a4-22ae-4196-b30e-4c79b9f2bd03 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:24:36.950622 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a7b37a-ecce-4c43-ade8-db420192ee81 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:24:36.951495 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a7b37a-ecce-4c43-ade8-db420192ee81 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:36.951618 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a7b37a-ecce-4c43-ade8-db420192ee81 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:36.951854 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a7b37a-ecce-4c43-ade8-db420192ee81 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:36.963546 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a7b37a-ecce-4c43-ade8-db420192ee81 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:24:36.965195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a7b37a-ecce-4c43-ade8-db420192ee81 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:24:36.981672 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1195/2377] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:24:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:24:42.502568 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf215974-74da-4ef3-bc63-4e1ab6d8271f None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:42.503029 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf215974-74da-4ef3-bc63-4e1ab6d8271f None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:42.503176 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf215974-74da-4ef3-bc63-4e1ab6d8271f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:42.535679 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-bf215974-74da-4ef3-bc63-4e1ab6d8271f None None] MFA Rules not processed for user `5ebc33b6c372434287b632d70f0a9e83`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:24:42.537057 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-bf215974-74da-4ef3-bc63-4e1ab6d8271f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:24:42.641923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1183/2378] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:24:42 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:24:42.644656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a7b37a-ecce-4c43-ade8-db420192ee81 service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:24:42.657181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a7b37a-ecce-4c43-ade8-db420192ee81 service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:24:42.670173 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a460fc80-6b84-40a6-a363-b37f7624fa09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:24:42.671118 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a460fc80-6b84-40a6-a363-b37f7624fa09 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:42.671503 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a460fc80-6b84-40a6-a363-b37f7624fa09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:42.671886 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a460fc80-6b84-40a6-a363-b37f7624fa09 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:42.672589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a460fc80-6b84-40a6-a363-b37f7624fa09 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:24:42.675055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a460fc80-6b84-40a6-a363-b37f7624fa09 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:24:42.719012 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1196/2379] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:24:42 2022] POST /identity/v3/projects => generated 371 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:24:42.719559 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf215974-74da-4ef3-bc63-4e1ab6d8271f None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:24:42.735152 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf215974-74da-4ef3-bc63-4e1ab6d8271f None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:24:42.758892 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:24:42.839380 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d193972f-7873-4b7a-a370-983cf7634973 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:24:42.841486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d193972f-7873-4b7a-a370-983cf7634973 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:42.841929 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d193972f-7873-4b7a-a370-983cf7634973 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:42.842667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d193972f-7873-4b7a-a370-983cf7634973 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:42.843513 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d193972f-7873-4b7a-a370-983cf7634973 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:24:42.846179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d193972f-7873-4b7a-a370-983cf7634973 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:24:42.901155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1184/2380] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:24:42 2022] POST /identity/v3/users => generated 292 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:24:42.905078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a460fc80-6b84-40a6-a363-b37f7624fa09 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:24:42.921807 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a460fc80-6b84-40a6-a363-b37f7624fa09 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:24:42.929555 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf417e85-f051-4f4e-8d68-69aef4898580 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:24:42.930221 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf417e85-f051-4f4e-8d68-69aef4898580 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:42.930499 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf417e85-f051-4f4e-8d68-69aef4898580 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:42.930772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf417e85-f051-4f4e-8d68-69aef4898580 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:42.931422 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf417e85-f051-4f4e-8d68-69aef4898580 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:24:42.934107 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf417e85-f051-4f4e-8d68-69aef4898580 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:24:42.949852 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1197/2381] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:24:42 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:24:42.953718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d193972f-7873-4b7a-a370-983cf7634973 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:24:42.966359 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d193972f-7873-4b7a-a370-983cf7634973 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:24:42.971396 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b909dc1b-67ef-447e-9324-c4fd34f84212 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:24:42.972077 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b909dc1b-67ef-447e-9324-c4fd34f84212 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:42.972359 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b909dc1b-67ef-447e-9324-c4fd34f84212 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:42.972653 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b909dc1b-67ef-447e-9324-c4fd34f84212 admin admin] PATH_INFO: `/v3/projects/4af948c97bfd43789719fb833d38e164/users/0beee66fb7a945d3a629b596634dd368/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:42.990259 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b909dc1b-67ef-447e-9324-c4fd34f84212 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4af948c97bfd43789719fb833d38e164, user_id=0beee66fb7a945d3a629b596634dd368, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:24:42.994426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b909dc1b-67ef-447e-9324-c4fd34f84212 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:24:43.014969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1185/2382] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:24:42 2022] PUT /identity/v3/projects/4af948c97bfd43789719fb833d38e164/users/0beee66fb7a945d3a629b596634dd368/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:24:43.018748 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf417e85-f051-4f4e-8d68-69aef4898580 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:24:43.029761 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf417e85-f051-4f4e-8d68-69aef4898580 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:24:43.090917 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e5704d-426e-416c-9fa2-e393297f6709 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:24:43.091532 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e5704d-426e-416c-9fa2-e393297f6709 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:43.091809 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e5704d-426e-416c-9fa2-e393297f6709 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:43.092077 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e5704d-426e-416c-9fa2-e393297f6709 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:43.092717 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e5704d-426e-416c-9fa2-e393297f6709 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:24:43.094411 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e5704d-426e-416c-9fa2-e393297f6709 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:24:43.109205 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1198/2383] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:24:43 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:24:43.110572 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b909dc1b-67ef-447e-9324-c4fd34f84212 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:24:43.123199 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b909dc1b-67ef-447e-9324-c4fd34f84212 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:24:43.128094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ca62df-3bb1-4bd8-bc73-25eb59f5010a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:24:43.128755 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ca62df-3bb1-4bd8-bc73-25eb59f5010a admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:43.129032 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ca62df-3bb1-4bd8-bc73-25eb59f5010a admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:43.129302 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ca62df-3bb1-4bd8-bc73-25eb59f5010a admin admin] PATH_INFO: `/v3/projects/4af948c97bfd43789719fb833d38e164/users/0beee66fb7a945d3a629b596634dd368/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:43.132565 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ca62df-3bb1-4bd8-bc73-25eb59f5010a admin admin] RBAC: Authorizing `identity:create_grant(project_id=4af948c97bfd43789719fb833d38e164, user_id=0beee66fb7a945d3a629b596634dd368, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:24:43.134896 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ca62df-3bb1-4bd8-bc73-25eb59f5010a admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:24:43.147548 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1186/2384] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:24:43 2022] PUT /identity/v3/projects/4af948c97bfd43789719fb833d38e164/users/0beee66fb7a945d3a629b596634dd368/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:24:46.497554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a077ad46-debf-4a1d-97a4-dea047ce527e None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:46.498405 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a077ad46-debf-4a1d-97a4-dea047ce527e None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:46.498829 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a077ad46-debf-4a1d-97a4-dea047ce527e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:46.525609 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-a077ad46-debf-4a1d-97a4-dea047ce527e None None] MFA Rules not processed for user `5a2fcc8c9ac44dceadcc30d163bbfa0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:24:46.527214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a077ad46-debf-4a1d-97a4-dea047ce527e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:24:46.571140 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:24:46.630714 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1199/2385] 10.222.0.26 () {58 vars in 967 bytes} [Tue Sep 27 04:24:46 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:24:46.699090 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ca62df-3bb1-4bd8-bc73-25eb59f5010a admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:24:46.701773 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-56ca62df-3bb1-4bd8-bc73-25eb59f5010a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:24:46.801365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ca62df-3bb1-4bd8-bc73-25eb59f5010a admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:24:46.817513 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf83b902-a1e1-4937-bd53-e00763f16048 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '181dd5dc720c4a97860852062b7b19c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:24:46.818338 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf83b902-a1e1-4937-bd53-e00763f16048 service swift] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:46.818620 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf83b902-a1e1-4937-bd53-e00763f16048 service swift] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:46.819630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf83b902-a1e1-4937-bd53-e00763f16048 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:46.841265 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf83b902-a1e1-4937-bd53-e00763f16048 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:24:46.842880 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf83b902-a1e1-4937-bd53-e00763f16048 service swift] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:24:46.863189 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1187/2386] 10.222.0.26 () {60 vars in 1384 bytes} [Tue Sep 27 04:24:46 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 04:24:48.932908 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8d729a-6fd0-46ca-b72e-b867bb485667 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:48.933441 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8d729a-6fd0-46ca-b72e-b867bb485667 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:48.933441 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8d729a-6fd0-46ca-b72e-b867bb485667 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:48.965968 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-bc8d729a-6fd0-46ca-b72e-b867bb485667 None None] MFA Rules not processed for user `0beee66fb7a945d3a629b596634dd368`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:24:48.967298 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-bc8d729a-6fd0-46ca-b72e-b867bb485667 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:24:49.091073 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1200/2387] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:24:48 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:24:49.108439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf83b902-a1e1-4937-bd53-e00763f16048 service swift] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:24:49.110530 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-bf83b902-a1e1-4937-bd53-e00763f16048 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:24:49.228081 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:24:49.261881 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf83b902-a1e1-4937-bd53-e00763f16048 service swift] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:24:49.281287 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c7d4db-25f6-4846-ad27-a817a04faaf9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:24:49.283014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c7d4db-25f6-4846-ad27-a817a04faaf9 service nova] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:49.283319 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c7d4db-25f6-4846-ad27-a817a04faaf9 service nova] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:49.283652 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c7d4db-25f6-4846-ad27-a817a04faaf9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:49.360073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c7d4db-25f6-4846-ad27-a817a04faaf9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:24:49.361861 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c7d4db-25f6-4846-ad27-a817a04faaf9 service nova] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:24:49.375103 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1188/2388] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:24:49 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 267 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:24:51.846058 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5b37e7-034e-47eb-ac5b-2b6ea88f2528 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:51.846964 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5b37e7-034e-47eb-ac5b-2b6ea88f2528 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:51.847539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5b37e7-034e-47eb-ac5b-2b6ea88f2528 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:51.890122 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-6b5b37e7-034e-47eb-ac5b-2b6ea88f2528 None None] MFA Rules not processed for user `5ebc33b6c372434287b632d70f0a9e83`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:24:51.891504 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6b5b37e7-034e-47eb-ac5b-2b6ea88f2528 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:24:51.905787 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1201/2389] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:24:51 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:24:52.819638 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b27124-b93f-4e2f-a379-f5ae2266911e None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:52.820146 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b27124-b93f-4e2f-a379-f5ae2266911e None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:52.820241 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b27124-b93f-4e2f-a379-f5ae2266911e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:52.840247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-c0b27124-b93f-4e2f-a379-f5ae2266911e None None] MFA Rules not processed for user `2d0ba2ae3ffe4bb9b99f1347719df411`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:24:52.842644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c0b27124-b93f-4e2f-a379-f5ae2266911e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:24:52.921699 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1189/2390] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:24:52 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:24:56.720861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb59013-f840-41b2-b890-9dc95c2de25a None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:56.721449 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb59013-f840-41b2-b890-9dc95c2de25a None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:56.721449 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb59013-f840-41b2-b890-9dc95c2de25a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:56.747808 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-1eb59013-f840-41b2-b890-9dc95c2de25a None None] MFA Rules not processed for user `f3438d850ce541cb9027ea3d5fe7dbf3`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:24:56.749969 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-1eb59013-f840-41b2-b890-9dc95c2de25a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:24:56.909891 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1202/2391] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:24:56 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:24:57.969689 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b27124-b93f-4e2f-a379-f5ae2266911e None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:24:57.972027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c0b27124-b93f-4e2f-a379-f5ae2266911e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:24:58.117700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b27124-b93f-4e2f-a379-f5ae2266911e None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:24:58.123737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90277a6-fbd8-478c-8f64-4dc24978391a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:24:58.123941 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90277a6-fbd8-478c-8f64-4dc24978391a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:58.124154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90277a6-fbd8-478c-8f64-4dc24978391a admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:58.124524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90277a6-fbd8-478c-8f64-4dc24978391a admin admin] PATH_INFO: `/v3/users/42d91d56296e4fdd917495df41401921` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:58.125693 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90277a6-fbd8-478c-8f64-4dc24978391a admin admin] RBAC: Authorizing `identity:delete_user(user_id=42d91d56296e4fdd917495df41401921)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:24:58.127648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90277a6-fbd8-478c-8f64-4dc24978391a admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:24:58.191988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:24:58.202475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-c90277a6-fbd8-478c-8f64-4dc24978391a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '42d91d56296e4fdd917495df41401921'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:58.213976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-c90277a6-fbd8-478c-8f64-4dc24978391a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '42d91d56296e4fdd917495df41401921'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:58.214968 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-c90277a6-fbd8-478c-8f64-4dc24978391a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '42d91d56296e4fdd917495df41401921'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:58.227934 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-c90277a6-fbd8-478c-8f64-4dc24978391a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '42d91d56296e4fdd917495df41401921'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:58.244812 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-c90277a6-fbd8-478c-8f64-4dc24978391a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '42d91d56296e4fdd917495df41401921'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:58.258062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1190/2392] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:24:57 2022] DELETE /identity/v3/users/42d91d56296e4fdd917495df41401921 => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:24:58.486161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb59013-f840-41b2-b890-9dc95c2de25a None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:24:58.500875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb59013-f840-41b2-b890-9dc95c2de25a None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:24:58.584723 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c75f41-61e6-4e3c-ac3f-5dfa6c43f0ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:24:58.585463 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c75f41-61e6-4e3c-ac3f-5dfa6c43f0ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:58.585796 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c75f41-61e6-4e3c-ac3f-5dfa6c43f0ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:58.586091 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c75f41-61e6-4e3c-ac3f-5dfa6c43f0ab admin admin] PATH_INFO: `/v3/projects/7e7b2ff0f0104769a550d0c0692ab713` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:58.587278 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c75f41-61e6-4e3c-ac3f-5dfa6c43f0ab admin admin] RBAC: Authorizing `identity:delete_project(project_id=7e7b2ff0f0104769a550d0c0692ab713)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:24:58.589253 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c75f41-61e6-4e3c-ac3f-5dfa6c43f0ab admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:24:58.597863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:24:58.656902 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-81c75f41-61e6-4e3c-ac3f-5dfa6c43f0ab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7e7b2ff0f0104769a550d0c0692ab713', 'request_id': 'req-81c75f41-61e6-4e3c-ac3f-5dfa6c43f0ab'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:58.667299 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-81c75f41-61e6-4e3c-ac3f-5dfa6c43f0ab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7e7b2ff0f0104769a550d0c0692ab713', 'request_id': 'req-81c75f41-61e6-4e3c-ac3f-5dfa6c43f0ab'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:58.674340 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-81c75f41-61e6-4e3c-ac3f-5dfa6c43f0ab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7e7b2ff0f0104769a550d0c0692ab713', 'request_id': 'req-81c75f41-61e6-4e3c-ac3f-5dfa6c43f0ab'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:58.691523 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-81c75f41-61e6-4e3c-ac3f-5dfa6c43f0ab admin admin] The token cache is being invalidate because project 7e7b2ff0f0104769a550d0c0692ab713 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:24:58.691987 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-81c75f41-61e6-4e3c-ac3f-5dfa6c43f0ab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:58.692734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-81c75f41-61e6-4e3c-ac3f-5dfa6c43f0ab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:58.695398 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1203/2393] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:24:58 2022] DELETE /identity/v3/projects/7e7b2ff0f0104769a550d0c0692ab713 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:24:58.699128 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90277a6-fbd8-478c-8f64-4dc24978391a admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:24:58.700684 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c90277a6-fbd8-478c-8f64-4dc24978391a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:24:58.719146 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90277a6-fbd8-478c-8f64-4dc24978391a admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:24:58.794053 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a2e28a-29b1-4f41-a1e8-6423db620e6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:24:58.794963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a2e28a-29b1-4f41-a1e8-6423db620e6e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:58.795343 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a2e28a-29b1-4f41-a1e8-6423db620e6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:58.795714 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a2e28a-29b1-4f41-a1e8-6423db620e6e admin admin] PATH_INFO: `/v3/users/01c2a355a3e8409a8bd7340cf5979036` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:58.796958 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a2e28a-29b1-4f41-a1e8-6423db620e6e admin admin] RBAC: Authorizing `identity:delete_user(user_id=01c2a355a3e8409a8bd7340cf5979036)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:24:58.799516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a2e28a-29b1-4f41-a1e8-6423db620e6e admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:24:58.860681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-05a2e28a-29b1-4f41-a1e8-6423db620e6e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '01c2a355a3e8409a8bd7340cf5979036'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:58.870216 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-05a2e28a-29b1-4f41-a1e8-6423db620e6e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '01c2a355a3e8409a8bd7340cf5979036'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:58.871010 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-05a2e28a-29b1-4f41-a1e8-6423db620e6e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '01c2a355a3e8409a8bd7340cf5979036'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:58.881792 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-05a2e28a-29b1-4f41-a1e8-6423db620e6e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '01c2a355a3e8409a8bd7340cf5979036'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:58.906612 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-05a2e28a-29b1-4f41-a1e8-6423db620e6e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '01c2a355a3e8409a8bd7340cf5979036'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:58.916072 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1191/2394] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:24:58 2022] DELETE /identity/v3/users/01c2a355a3e8409a8bd7340cf5979036 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:24:59.131244 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c75f41-61e6-4e3c-ac3f-5dfa6c43f0ab admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:24:59.139098 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-81c75f41-61e6-4e3c-ac3f-5dfa6c43f0ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:24:59.159115 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c75f41-61e6-4e3c-ac3f-5dfa6c43f0ab admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:24:59.263047 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044d64f4-c921-4f34-bb36-56be0f218ec0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:24:59.263691 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044d64f4-c921-4f34-bb36-56be0f218ec0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:59.263968 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044d64f4-c921-4f34-bb36-56be0f218ec0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:59.264239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044d64f4-c921-4f34-bb36-56be0f218ec0 admin admin] PATH_INFO: `/v3/projects/3030064c1315435699a7e7d5a831a5b2` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:59.265382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044d64f4-c921-4f34-bb36-56be0f218ec0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3030064c1315435699a7e7d5a831a5b2)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:24:59.267302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044d64f4-c921-4f34-bb36-56be0f218ec0 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:24:59.322984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-044d64f4-c921-4f34-bb36-56be0f218ec0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3030064c1315435699a7e7d5a831a5b2', 'request_id': 'req-044d64f4-c921-4f34-bb36-56be0f218ec0'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:59.329138 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-044d64f4-c921-4f34-bb36-56be0f218ec0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3030064c1315435699a7e7d5a831a5b2', 'request_id': 'req-044d64f4-c921-4f34-bb36-56be0f218ec0'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:59.336070 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-044d64f4-c921-4f34-bb36-56be0f218ec0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3030064c1315435699a7e7d5a831a5b2', 'request_id': 'req-044d64f4-c921-4f34-bb36-56be0f218ec0'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:59.354265 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-044d64f4-c921-4f34-bb36-56be0f218ec0 admin admin] The token cache is being invalidate because project 3030064c1315435699a7e7d5a831a5b2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:24:59.354265 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-044d64f4-c921-4f34-bb36-56be0f218ec0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:59.354265 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-044d64f4-c921-4f34-bb36-56be0f218ec0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:24:59.354265 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1204/2395] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:24:59 2022] DELETE /identity/v3/projects/3030064c1315435699a7e7d5a831a5b2 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:24:59.414718 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a2e28a-29b1-4f41-a1e8-6423db620e6e admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:24:59.416814 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-05a2e28a-29b1-4f41-a1e8-6423db620e6e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:24:59.486975 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:24:59.526885 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a2e28a-29b1-4f41-a1e8-6423db620e6e admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:24:59.632892 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d9a227-ddcd-486c-8e81-bcdc9df1f518 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:24:59.633525 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d9a227-ddcd-486c-8e81-bcdc9df1f518 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:24:59.633853 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d9a227-ddcd-486c-8e81-bcdc9df1f518 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:24:59.634130 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d9a227-ddcd-486c-8e81-bcdc9df1f518 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:24:59.635897 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-d0d9a227-ddcd-486c-8e81-bcdc9df1f518 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:24:59.799660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d9a227-ddcd-486c-8e81-bcdc9df1f518 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:24:59.801530 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d9a227-ddcd-486c-8e81-bcdc9df1f518 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:24:59.817838 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1192/2396] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:24:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:25:02.144203 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044d64f4-c921-4f34-bb36-56be0f218ec0 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:25:02.145799 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-044d64f4-c921-4f34-bb36-56be0f218ec0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:25:02.158554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044d64f4-c921-4f34-bb36-56be0f218ec0 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:25:02.162525 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1772263e-17e3-4a4b-9b38-107145f29000 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:25:02.163146 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1772263e-17e3-4a4b-9b38-107145f29000 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:25:02.163422 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1772263e-17e3-4a4b-9b38-107145f29000 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:25:02.163691 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1772263e-17e3-4a4b-9b38-107145f29000 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:25:02.165454 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-1772263e-17e3-4a4b-9b38-107145f29000 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:25:02.180164 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1772263e-17e3-4a4b-9b38-107145f29000 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:25:02.181708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1772263e-17e3-4a4b-9b38-107145f29000 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:25:02.192411 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1205/2397] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:25:02 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:25:05.920542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d9a227-ddcd-486c-8e81-bcdc9df1f518 service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:25:05.922022 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-d0d9a227-ddcd-486c-8e81-bcdc9df1f518 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:25:05.936058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d9a227-ddcd-486c-8e81-bcdc9df1f518 service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:25:05.941841 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233a19da-b316-473d-9631-39dce9ff038c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:25:05.942460 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233a19da-b316-473d-9631-39dce9ff038c service nova] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:25:05.942740 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233a19da-b316-473d-9631-39dce9ff038c service nova] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:25:05.943020 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233a19da-b316-473d-9631-39dce9ff038c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:25:05.944672 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-233a19da-b316-473d-9631-39dce9ff038c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:25:06.003188 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233a19da-b316-473d-9631-39dce9ff038c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:25:06.005055 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233a19da-b316-473d-9631-39dce9ff038c service nova] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:25:06.015967 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1193/2398] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:25:05 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:25:12.187632 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=78,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:25:16.010982 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=85 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:25:23.254362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1772263e-17e3-4a4b-9b38-107145f29000 service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:25:23.255838 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-1772263e-17e3-4a4b-9b38-107145f29000 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:25:23.271748 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1772263e-17e3-4a4b-9b38-107145f29000 service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:25:23.275886 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79513d0a-a714-4878-bae9-58fd9fe4779d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:25:23.276461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79513d0a-a714-4878-bae9-58fd9fe4779d service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:25:23.276732 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79513d0a-a714-4878-bae9-58fd9fe4779d service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:25:23.276998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79513d0a-a714-4878-bae9-58fd9fe4779d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:25:23.278704 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-79513d0a-a714-4878-bae9-58fd9fe4779d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:25:23.287974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79513d0a-a714-4878-bae9-58fd9fe4779d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:25:23.289968 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79513d0a-a714-4878-bae9-58fd9fe4779d service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:25:23.297624 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1206/2399] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:25:23 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:25:25.936785 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233a19da-b316-473d-9631-39dce9ff038c service nova] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:25:25.938157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-233a19da-b316-473d-9631-39dce9ff038c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:25:25.952693 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233a19da-b316-473d-9631-39dce9ff038c service nova] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:25:25.958127 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f259f462-978a-4456-956b-4042f41d61b6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:25:25.958712 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f259f462-978a-4456-956b-4042f41d61b6 service nova] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:25:25.958992 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f259f462-978a-4456-956b-4042f41d61b6 service nova] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:25:25.959266 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f259f462-978a-4456-956b-4042f41d61b6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:25:25.961539 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-f259f462-978a-4456-956b-4042f41d61b6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:25:26.018773 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f259f462-978a-4456-956b-4042f41d61b6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:25:26.020347 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f259f462-978a-4456-956b-4042f41d61b6 service nova] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:25:26.030260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1194/2400] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:25:25 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:25:26.350114 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79513d0a-a714-4878-bae9-58fd9fe4779d service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:25:26.357441 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79513d0a-a714-4878-bae9-58fd9fe4779d service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:25:26.361331 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6bc06b6-ff2f-4ee4-9495-743f45c64c52 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:25:26.362096 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bc06b6-ff2f-4ee4-9495-743f45c64c52 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:25:26.362368 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bc06b6-ff2f-4ee4-9495-743f45c64c52 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:25:26.362633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bc06b6-ff2f-4ee4-9495-743f45c64c52 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:25:26.364293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d6bc06b6-ff2f-4ee4-9495-743f45c64c52 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:25:26.431716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6bc06b6-ff2f-4ee4-9495-743f45c64c52 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:25:26.433142 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6bc06b6-ff2f-4ee4-9495-743f45c64c52 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:25:26.442460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1207/2401] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:25:26 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:25:36.025881 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=28 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:25:36.438208 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=36 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:25:43.122053 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f259f462-978a-4456-956b-4042f41d61b6 service nova] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:25:43.133357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f259f462-978a-4456-956b-4042f41d61b6 service nova] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:25:43.139961 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07da97d2-e7cc-4c50-b963-d2f8abd28c42 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:25:43.140386 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07da97d2-e7cc-4c50-b963-d2f8abd28c42 service nova] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:25:43.140671 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07da97d2-e7cc-4c50-b963-d2f8abd28c42 service nova] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:25:43.140959 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07da97d2-e7cc-4c50-b963-d2f8abd28c42 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:25:43.142828 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-07da97d2-e7cc-4c50-b963-d2f8abd28c42 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:25:43.156824 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07da97d2-e7cc-4c50-b963-d2f8abd28c42 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:25:43.158452 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07da97d2-e7cc-4c50-b963-d2f8abd28c42 service nova] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:25:43.184285 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1195/2402] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:25:43 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:25:46.473109 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b24dfd-c587-4563-93a1-d9171edf22fc None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:25:46.473591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b24dfd-c587-4563-93a1-d9171edf22fc None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:25:46.473695 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b24dfd-c587-4563-93a1-d9171edf22fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:25:46.494061 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-90b24dfd-c587-4563-93a1-d9171edf22fc None None] MFA Rules not processed for user `5a2fcc8c9ac44dceadcc30d163bbfa0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:25:46.495256 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-90b24dfd-c587-4563-93a1-d9171edf22fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:25:46.548472 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1208/2403] 10.222.0.26 () {58 vars in 1009 bytes} [Tue Sep 27 04:25:46 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:25:46.570744 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07da97d2-e7cc-4c50-b963-d2f8abd28c42 service nova] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:25:46.572764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-07da97d2-e7cc-4c50-b963-d2f8abd28c42 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:25:46.642062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07da97d2-e7cc-4c50-b963-d2f8abd28c42 service nova] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:25:46.646924 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52447893-df6d-4d20-b535-57319166b385 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '181dd5dc720c4a97860852062b7b19c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:25:46.647550 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52447893-df6d-4d20-b535-57319166b385 service swift] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:25:46.647830 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52447893-df6d-4d20-b535-57319166b385 service swift] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:25:46.648103 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52447893-df6d-4d20-b535-57319166b385 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:25:46.657699 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52447893-df6d-4d20-b535-57319166b385 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:25:46.659321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52447893-df6d-4d20-b535-57319166b385 service swift] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:25:46.667832 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1196/2404] 10.222.0.26 () {60 vars in 1384 bytes} [Tue Sep 27 04:25:46 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 04:25:47.172348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b24dfd-c587-4563-93a1-d9171edf22fc None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:25:47.202502 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b24dfd-c587-4563-93a1-d9171edf22fc None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:25:47.218893 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28cdf00-f27a-4fdc-ac16-3518a922ba1f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:25:47.220812 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28cdf00-f27a-4fdc-ac16-3518a922ba1f service nova] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:25:47.221253 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28cdf00-f27a-4fdc-ac16-3518a922ba1f service nova] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:25:47.221691 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28cdf00-f27a-4fdc-ac16-3518a922ba1f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:25:47.223759 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d28cdf00-f27a-4fdc-ac16-3518a922ba1f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:25:47.241072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28cdf00-f27a-4fdc-ac16-3518a922ba1f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:25:47.242916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28cdf00-f27a-4fdc-ac16-3518a922ba1f service nova] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:25:47.257799 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1209/2405] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:25:47 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:25:56.664539 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=38 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:25:57.251629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=31 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:00.648379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52447893-df6d-4d20-b535-57319166b385 service swift] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:00.650466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-52447893-df6d-4d20-b535-57319166b385 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:00.731302 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52447893-df6d-4d20-b535-57319166b385 service swift] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:00.736664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fdfb62-8fd2-4e24-8313-b778911e7e0d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3a1bc198dfc0486e9e56d4fb28cda05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:00.737530 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fdfb62-8fd2-4e24-8313-b778911e7e0d service placement] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:00.738028 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fdfb62-8fd2-4e24-8313-b778911e7e0d service placement] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:00.738331 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fdfb62-8fd2-4e24-8313-b778911e7e0d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:00.740368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-f3fdfb62-8fd2-4e24-8313-b778911e7e0d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:00.754342 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fdfb62-8fd2-4e24-8313-b778911e7e0d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:00.756574 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fdfb62-8fd2-4e24-8313-b778911e7e0d service placement] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:00.767560 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1197/2406] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:26:00 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:26:02.659249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87d5a46-026c-4319-b5f9-093b7bd3d3e6 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:02.659804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87d5a46-026c-4319-b5f9-093b7bd3d3e6 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:02.659804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87d5a46-026c-4319-b5f9-093b7bd3d3e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:02.680641 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-c87d5a46-026c-4319-b5f9-093b7bd3d3e6 None None] MFA Rules not processed for user `5a2fcc8c9ac44dceadcc30d163bbfa0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:02.682006 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c87d5a46-026c-4319-b5f9-093b7bd3d3e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:02.692519 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1210/2407] 10.222.0.26 () {58 vars in 1009 bytes} [Tue Sep 27 04:26:02 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:02.709799 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fdfb62-8fd2-4e24-8313-b778911e7e0d service placement] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:02.730025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fdfb62-8fd2-4e24-8313-b778911e7e0d service placement] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:02.740065 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24ebb95-bb71-4d63-b678-1ac8261f2f39 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '181dd5dc720c4a97860852062b7b19c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:02.740991 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24ebb95-bb71-4d63-b678-1ac8261f2f39 service swift] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:02.741379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24ebb95-bb71-4d63-b678-1ac8261f2f39 service swift] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:02.741818 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24ebb95-bb71-4d63-b678-1ac8261f2f39 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:02.761529 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24ebb95-bb71-4d63-b678-1ac8261f2f39 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:02.765897 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24ebb95-bb71-4d63-b678-1ac8261f2f39 service swift] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:02.776357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1198/2408] 10.222.0.26 () {60 vars in 1384 bytes} [Tue Sep 27 04:26:02 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 04:26:03.057338 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c87d5a46-026c-4319-b5f9-093b7bd3d3e6 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:03.058798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c87d5a46-026c-4319-b5f9-093b7bd3d3e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:03.231328 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c87d5a46-026c-4319-b5f9-093b7bd3d3e6 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:03.236670 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b4c5af-e43a-4e13-8449-91df162d1650 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:03.237315 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b4c5af-e43a-4e13-8449-91df162d1650 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:03.237595 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b4c5af-e43a-4e13-8449-91df162d1650 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:03.240654 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b4c5af-e43a-4e13-8449-91df162d1650 admin admin] PATH_INFO: `/v3/users/2d0ba2ae3ffe4bb9b99f1347719df411` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:03.244915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b4c5af-e43a-4e13-8449-91df162d1650 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2d0ba2ae3ffe4bb9b99f1347719df411)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:03.246839 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b4c5af-e43a-4e13-8449-91df162d1650 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:03.346901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-11b4c5af-e43a-4e13-8449-91df162d1650 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2d0ba2ae3ffe4bb9b99f1347719df411'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:03.363723 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-11b4c5af-e43a-4e13-8449-91df162d1650 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2d0ba2ae3ffe4bb9b99f1347719df411'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:03.377769 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-11b4c5af-e43a-4e13-8449-91df162d1650 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2d0ba2ae3ffe4bb9b99f1347719df411'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:03.385188 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-11b4c5af-e43a-4e13-8449-91df162d1650 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2d0ba2ae3ffe4bb9b99f1347719df411'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:03.404915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-11b4c5af-e43a-4e13-8449-91df162d1650 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2d0ba2ae3ffe4bb9b99f1347719df411'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:03.411875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1211/2409] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:26:03 2022] DELETE /identity/v3/users/2d0ba2ae3ffe4bb9b99f1347719df411 => generated 0 bytes in 355 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:03.626275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24ebb95-bb71-4d63-b678-1ac8261f2f39 service swift] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:03.650133 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24ebb95-bb71-4d63-b678-1ac8261f2f39 service swift] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:03.743537 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7449e726-8cbd-4e7e-bbc8-036d369fd376 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:03.744884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7449e726-8cbd-4e7e-bbc8-036d369fd376 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:03.745323 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7449e726-8cbd-4e7e-bbc8-036d369fd376 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:03.745810 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7449e726-8cbd-4e7e-bbc8-036d369fd376 admin admin] PATH_INFO: `/v3/projects/c25f11f92d4e4fd8965fe630ad77f3c3` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:03.747398 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7449e726-8cbd-4e7e-bbc8-036d369fd376 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c25f11f92d4e4fd8965fe630ad77f3c3)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:03.750629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7449e726-8cbd-4e7e-bbc8-036d369fd376 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:03.834174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-7449e726-8cbd-4e7e-bbc8-036d369fd376 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c25f11f92d4e4fd8965fe630ad77f3c3', 'request_id': 'req-7449e726-8cbd-4e7e-bbc8-036d369fd376'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:03.842801 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-7449e726-8cbd-4e7e-bbc8-036d369fd376 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c25f11f92d4e4fd8965fe630ad77f3c3', 'request_id': 'req-7449e726-8cbd-4e7e-bbc8-036d369fd376'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:03.849662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-7449e726-8cbd-4e7e-bbc8-036d369fd376 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c25f11f92d4e4fd8965fe630ad77f3c3', 'request_id': 'req-7449e726-8cbd-4e7e-bbc8-036d369fd376'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:03.872532 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-7449e726-8cbd-4e7e-bbc8-036d369fd376 admin admin] The token cache is being invalidate because project c25f11f92d4e4fd8965fe630ad77f3c3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:26:03.873142 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-7449e726-8cbd-4e7e-bbc8-036d369fd376 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:03.874029 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-7449e726-8cbd-4e7e-bbc8-036d369fd376 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:03.875766 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1199/2410] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:26:03 2022] DELETE /identity/v3/projects/c25f11f92d4e4fd8965fe630ad77f3c3 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:03.883228 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c6cd36-1cca-4540-a95a-1116849e1f4d None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:03.885116 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c6cd36-1cca-4540-a95a-1116849e1f4d None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:03.885637 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c6cd36-1cca-4540-a95a-1116849e1f4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:03.923134 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-95c6cd36-1cca-4540-a95a-1116849e1f4d None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:03.924363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-95c6cd36-1cca-4540-a95a-1116849e1f4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:04.002670 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1212/2411] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:04.010912 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf031601-cdc7-4610-ad5c-32516231016e None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:04.011199 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf031601-cdc7-4610-ad5c-32516231016e None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:04.011475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf031601-cdc7-4610-ad5c-32516231016e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:04.026834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:04.041556 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-cf031601-cdc7-4610-ad5c-32516231016e None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:04.043606 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-cf031601-cdc7-4610-ad5c-32516231016e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:04.052854 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1200/2412] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:04.058632 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c6cd36-1cca-4540-a95a-1116849e1f4d None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:04.074371 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c6cd36-1cca-4540-a95a-1116849e1f4d None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:04.080299 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57386eb-4636-4ef4-9987-f82b8c64e6b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:04.080919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57386eb-4636-4ef4-9987-f82b8c64e6b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:04.081232 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57386eb-4636-4ef4-9987-f82b8c64e6b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:04.081496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57386eb-4636-4ef4-9987-f82b8c64e6b4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:04.082192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57386eb-4636-4ef4-9987-f82b8c64e6b4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:26:04.082491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57386eb-4636-4ef4-9987-f82b8c64e6b4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:04.084088 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57386eb-4636-4ef4-9987-f82b8c64e6b4 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:04.106326 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:04.117163 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf031601-cdc7-4610-ad5c-32516231016e None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:04.120033 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1213/2413] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:26:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:26:04.138309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf031601-cdc7-4610-ad5c-32516231016e None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:04.143920 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6cdcd1-dc8a-418b-a78a-b103b125e8bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:04.144539 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6cdcd1-dc8a-418b-a78a-b103b125e8bb admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:04.144843 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6cdcd1-dc8a-418b-a78a-b103b125e8bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:04.145116 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6cdcd1-dc8a-418b-a78a-b103b125e8bb admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:04.145782 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6cdcd1-dc8a-418b-a78a-b103b125e8bb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:04.147680 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6cdcd1-dc8a-418b-a78a-b103b125e8bb admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:04.180066 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1201/2414] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:26:04 2022] POST /identity/v3/projects => generated 385 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:26:04.181896 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57386eb-4636-4ef4-9987-f82b8c64e6b4 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:04.191958 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57386eb-4636-4ef4-9987-f82b8c64e6b4 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:04.257922 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91a9a786-ef48-4635-b635-022c190151ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:04.258572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a9a786-ef48-4635-b635-022c190151ee admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:04.258852 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a9a786-ef48-4635-b635-022c190151ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:04.259127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a9a786-ef48-4635-b635-022c190151ee admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:04.259770 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91a9a786-ef48-4635-b635-022c190151ee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:04.261675 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91a9a786-ef48-4635-b635-022c190151ee admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:04.284169 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1214/2415] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:26:04 2022] POST /identity/v3/users => generated 293 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:26:04.294201 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6cdcd1-dc8a-418b-a78a-b103b125e8bb admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:04.305503 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6cdcd1-dc8a-418b-a78a-b103b125e8bb admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:04.310704 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ddc39a-664a-4511-a5bf-8dfe91ac9d77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:04.311300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ddc39a-664a-4511-a5bf-8dfe91ac9d77 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:04.311691 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ddc39a-664a-4511-a5bf-8dfe91ac9d77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:04.311979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ddc39a-664a-4511-a5bf-8dfe91ac9d77 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:04.312631 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ddc39a-664a-4511-a5bf-8dfe91ac9d77 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:04.314451 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ddc39a-664a-4511-a5bf-8dfe91ac9d77 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:04.331686 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1202/2416] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:04 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:04.336112 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91a9a786-ef48-4635-b635-022c190151ee admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:04.347878 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91a9a786-ef48-4635-b635-022c190151ee admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:04.353806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ccf294-35ca-4523-8759-259edafba815 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:04.354406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ccf294-35ca-4523-8759-259edafba815 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:04.354678 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ccf294-35ca-4523-8759-259edafba815 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:04.355006 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ccf294-35ca-4523-8759-259edafba815 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:04.355734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ccf294-35ca-4523-8759-259edafba815 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:04.357624 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ccf294-35ca-4523-8759-259edafba815 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:04.370485 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1215/2417] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:04 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:04.374270 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ddc39a-664a-4511-a5bf-8dfe91ac9d77 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:04.384898 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ddc39a-664a-4511-a5bf-8dfe91ac9d77 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:04.390142 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec34879e-6668-4ec6-9b8b-6259a9e962d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:04.391050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec34879e-6668-4ec6-9b8b-6259a9e962d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:04.391429 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec34879e-6668-4ec6-9b8b-6259a9e962d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:04.391801 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec34879e-6668-4ec6-9b8b-6259a9e962d1 admin admin] PATH_INFO: `/v3/projects/e2356c8895ab41f68983fc8328fad8cc/users/bcbf6702a9064ca48776744cce268284/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:04.420528 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec34879e-6668-4ec6-9b8b-6259a9e962d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2356c8895ab41f68983fc8328fad8cc, user_id=bcbf6702a9064ca48776744cce268284, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:04.423584 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec34879e-6668-4ec6-9b8b-6259a9e962d1 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:04.460229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1203/2418] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:26:04 2022] PUT /identity/v3/projects/e2356c8895ab41f68983fc8328fad8cc/users/bcbf6702a9064ca48776744cce268284/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:04.472619 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73a9959-4398-4bde-86dc-9ebc2b7b76ff None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:04.472901 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73a9959-4398-4bde-86dc-9ebc2b7b76ff None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:04.473180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73a9959-4398-4bde-86dc-9ebc2b7b76ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:04.502656 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-a73a9959-4398-4bde-86dc-9ebc2b7b76ff None None] MFA Rules not processed for user `bcbf6702a9064ca48776744cce268284`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:04.504009 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a73a9959-4398-4bde-86dc-9ebc2b7b76ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:04.590930 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1216/2419] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:04 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:04.602946 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec34879e-6668-4ec6-9b8b-6259a9e962d1 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:04.608078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ec34879e-6668-4ec6-9b8b-6259a9e962d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:04.704653 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec34879e-6668-4ec6-9b8b-6259a9e962d1 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:04.710155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce80af4a-21d5-4f3a-a2f0-a963dd47776f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:04.710758 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce80af4a-21d5-4f3a-a2f0-a963dd47776f service nova] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:04.711034 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce80af4a-21d5-4f3a-a2f0-a963dd47776f service nova] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:04.711305 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce80af4a-21d5-4f3a-a2f0-a963dd47776f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:04.719977 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce80af4a-21d5-4f3a-a2f0-a963dd47776f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:04.721566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce80af4a-21d5-4f3a-a2f0-a963dd47776f service nova] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:04.730734 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1204/2420] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:26:04 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:26:13.194851 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73a9959-4398-4bde-86dc-9ebc2b7b76ff None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:13.254687 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73a9959-4398-4bde-86dc-9ebc2b7b76ff None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:13.259172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834af0ca-740d-4f1e-a8e8-0293737e59f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:13.260303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834af0ca-740d-4f1e-a8e8-0293737e59f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:13.260581 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834af0ca-740d-4f1e-a8e8-0293737e59f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:13.260854 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834af0ca-740d-4f1e-a8e8-0293737e59f3 admin admin] PATH_INFO: `/v3/users/bcbf6702a9064ca48776744cce268284` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:13.261989 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-834af0ca-740d-4f1e-a8e8-0293737e59f3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bcbf6702a9064ca48776744cce268284)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:13.263899 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-834af0ca-740d-4f1e-a8e8-0293737e59f3 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:13.275126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:13.317869 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-834af0ca-740d-4f1e-a8e8-0293737e59f3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bcbf6702a9064ca48776744cce268284'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:13.326370 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-834af0ca-740d-4f1e-a8e8-0293737e59f3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bcbf6702a9064ca48776744cce268284'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:13.327068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-834af0ca-740d-4f1e-a8e8-0293737e59f3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bcbf6702a9064ca48776744cce268284'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:13.335702 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-834af0ca-740d-4f1e-a8e8-0293737e59f3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bcbf6702a9064ca48776744cce268284'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:13.349347 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-834af0ca-740d-4f1e-a8e8-0293737e59f3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bcbf6702a9064ca48776744cce268284'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:13.355624 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1217/2421] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:26:13 2022] DELETE /identity/v3/users/bcbf6702a9064ca48776744cce268284 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:13.360064 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9a00b2-9a1c-415e-8b25-b3e33c6a45fa None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:13.360549 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9a00b2-9a1c-415e-8b25-b3e33c6a45fa None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:13.360654 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9a00b2-9a1c-415e-8b25-b3e33c6a45fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:13.381467 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-bc9a00b2-9a1c-415e-8b25-b3e33c6a45fa None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:13.382756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-bc9a00b2-9a1c-415e-8b25-b3e33c6a45fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:13.445130 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1205/2422] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:13.454780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834af0ca-740d-4f1e-a8e8-0293737e59f3 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:13.456194 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-834af0ca-740d-4f1e-a8e8-0293737e59f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:13.513428 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834af0ca-740d-4f1e-a8e8-0293737e59f3 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:13.516981 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8fc8f8-59b1-4a4d-8c1c-7f52d99d28e4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:13.517577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8fc8f8-59b1-4a4d-8c1c-7f52d99d28e4 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:13.517881 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8fc8f8-59b1-4a4d-8c1c-7f52d99d28e4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:13.518152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8fc8f8-59b1-4a4d-8c1c-7f52d99d28e4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:13.526861 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8fc8f8-59b1-4a4d-8c1c-7f52d99d28e4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:13.528270 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8fc8f8-59b1-4a4d-8c1c-7f52d99d28e4 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:13.537277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1218/2423] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:26:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:26:13.799471 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9a00b2-9a1c-415e-8b25-b3e33c6a45fa None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:13.801173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-bc9a00b2-9a1c-415e-8b25-b3e33c6a45fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:13.813761 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9a00b2-9a1c-415e-8b25-b3e33c6a45fa None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:13.818282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b02346-6aaa-4554-a057-1b7b66a1d336 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:13.818903 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b02346-6aaa-4554-a057-1b7b66a1d336 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:13.819195 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b02346-6aaa-4554-a057-1b7b66a1d336 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:13.819466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b02346-6aaa-4554-a057-1b7b66a1d336 admin admin] PATH_INFO: `/v3/projects/e2356c8895ab41f68983fc8328fad8cc` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:13.820574 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b02346-6aaa-4554-a057-1b7b66a1d336 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e2356c8895ab41f68983fc8328fad8cc)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:13.823099 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b02346-6aaa-4554-a057-1b7b66a1d336 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:13.847382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:INSERT=2,keystone:DELETE=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:13.872957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-79b02346-6aaa-4554-a057-1b7b66a1d336 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e2356c8895ab41f68983fc8328fad8cc', 'request_id': 'req-79b02346-6aaa-4554-a057-1b7b66a1d336'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:13.879715 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-79b02346-6aaa-4554-a057-1b7b66a1d336 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e2356c8895ab41f68983fc8328fad8cc', 'request_id': 'req-79b02346-6aaa-4554-a057-1b7b66a1d336'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:13.887978 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-79b02346-6aaa-4554-a057-1b7b66a1d336 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e2356c8895ab41f68983fc8328fad8cc', 'request_id': 'req-79b02346-6aaa-4554-a057-1b7b66a1d336'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:13.905720 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-79b02346-6aaa-4554-a057-1b7b66a1d336 admin admin] The token cache is being invalidate because project e2356c8895ab41f68983fc8328fad8cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:26:13.905976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-79b02346-6aaa-4554-a057-1b7b66a1d336 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:13.906860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-79b02346-6aaa-4554-a057-1b7b66a1d336 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:13.908876 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1206/2424] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:26:13 2022] DELETE /identity/v3/projects/e2356c8895ab41f68983fc8328fad8cc => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:13.915397 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47e7b22-cf8f-4258-b1e4-f4b610165633 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:13.916311 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47e7b22-cf8f-4258-b1e4-f4b610165633 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:13.916654 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47e7b22-cf8f-4258-b1e4-f4b610165633 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:13.943260 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-f47e7b22-cf8f-4258-b1e4-f4b610165633 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:13.944456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-f47e7b22-cf8f-4258-b1e4-f4b610165633 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:14.011065 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1219/2425] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:14.019594 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb4eb00-0755-4b84-ac8a-71a32ef04ddd None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:14.019885 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb4eb00-0755-4b84-ac8a-71a32ef04ddd None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:14.020166 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb4eb00-0755-4b84-ac8a-71a32ef04ddd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:14.042699 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-7eb4eb00-0755-4b84-ac8a-71a32ef04ddd None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:14.043920 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-7eb4eb00-0755-4b84-ac8a-71a32ef04ddd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:14.052091 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1207/2426] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:14.057205 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47e7b22-cf8f-4258-b1e4-f4b610165633 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:14.067482 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47e7b22-cf8f-4258-b1e4-f4b610165633 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:14.073649 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83ef5e0-73b9-405e-82ae-48063e94732d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:14.074528 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83ef5e0-73b9-405e-82ae-48063e94732d admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:14.074925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83ef5e0-73b9-405e-82ae-48063e94732d admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:14.075186 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83ef5e0-73b9-405e-82ae-48063e94732d admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:14.075832 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83ef5e0-73b9-405e-82ae-48063e94732d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:26:14.076130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83ef5e0-73b9-405e-82ae-48063e94732d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:14.077870 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83ef5e0-73b9-405e-82ae-48063e94732d admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:14.090670 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1220/2427] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:26:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:26:14.094986 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb4eb00-0755-4b84-ac8a-71a32ef04ddd None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:14.105210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb4eb00-0755-4b84-ac8a-71a32ef04ddd None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:14.112281 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c868dfa-2093-489b-a4e9-b38ed9134251 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:14.112887 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c868dfa-2093-489b-a4e9-b38ed9134251 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:14.113162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c868dfa-2093-489b-a4e9-b38ed9134251 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:14.113429 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c868dfa-2093-489b-a4e9-b38ed9134251 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:14.114108 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c868dfa-2093-489b-a4e9-b38ed9134251 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:14.116036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c868dfa-2093-489b-a4e9-b38ed9134251 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:14.146655 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1208/2428] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:26:14 2022] POST /identity/v3/projects => generated 375 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:26:14.152246 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83ef5e0-73b9-405e-82ae-48063e94732d admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:14.163818 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83ef5e0-73b9-405e-82ae-48063e94732d admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:14.178690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:14.226056 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3d515b-5534-464a-931a-8554878c64a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:14.226718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3d515b-5534-464a-931a-8554878c64a6 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:14.226925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3d515b-5534-464a-931a-8554878c64a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:14.227200 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3d515b-5534-464a-931a-8554878c64a6 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:14.227818 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3d515b-5534-464a-931a-8554878c64a6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:14.229816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3d515b-5534-464a-931a-8554878c64a6 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:14.249726 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1221/2429] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:26:14 2022] POST /identity/v3/users => generated 288 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:26:14.254023 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c868dfa-2093-489b-a4e9-b38ed9134251 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:14.264726 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c868dfa-2093-489b-a4e9-b38ed9134251 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:14.269318 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6beb5b05-5834-4063-aaa2-c597ef29f6d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:14.269939 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beb5b05-5834-4063-aaa2-c597ef29f6d2 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:14.270222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beb5b05-5834-4063-aaa2-c597ef29f6d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:14.270495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beb5b05-5834-4063-aaa2-c597ef29f6d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:14.271134 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6beb5b05-5834-4063-aaa2-c597ef29f6d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:14.272703 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6beb5b05-5834-4063-aaa2-c597ef29f6d2 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:14.283050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1209/2430] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:14.286941 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3d515b-5534-464a-931a-8554878c64a6 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:14.297033 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3d515b-5534-464a-931a-8554878c64a6 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:14.301652 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46cf5927-1364-473a-8167-323895b6daae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:14.302272 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cf5927-1364-473a-8167-323895b6daae admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:14.302551 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cf5927-1364-473a-8167-323895b6daae admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:14.302824 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cf5927-1364-473a-8167-323895b6daae admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:14.303461 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46cf5927-1364-473a-8167-323895b6daae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:14.305069 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46cf5927-1364-473a-8167-323895b6daae admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:14.315497 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1222/2431] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:14 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:14.319386 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6beb5b05-5834-4063-aaa2-c597ef29f6d2 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:14.330138 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6beb5b05-5834-4063-aaa2-c597ef29f6d2 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:14.334751 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61c13ab-916a-4c7b-bd97-7c3673b5dc37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:14.335391 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61c13ab-916a-4c7b-bd97-7c3673b5dc37 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:14.335674 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61c13ab-916a-4c7b-bd97-7c3673b5dc37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:14.335949 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61c13ab-916a-4c7b-bd97-7c3673b5dc37 admin admin] PATH_INFO: `/v3/projects/49ef5f872e954a9295f1d2e1a00632b8/users/a293bc416349457ea83527a1ef42511f/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:14.350768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61c13ab-916a-4c7b-bd97-7c3673b5dc37 admin admin] RBAC: Authorizing `identity:create_grant(project_id=49ef5f872e954a9295f1d2e1a00632b8, user_id=a293bc416349457ea83527a1ef42511f, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:14.353389 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61c13ab-916a-4c7b-bd97-7c3673b5dc37 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:14.366624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1210/2432] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:26:14 2022] PUT /identity/v3/projects/49ef5f872e954a9295f1d2e1a00632b8/users/a293bc416349457ea83527a1ef42511f/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:26:14.377936 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02e5ec2-353d-46c0-bb1b-097b8704be07 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:14.378224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02e5ec2-353d-46c0-bb1b-097b8704be07 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:14.378504 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02e5ec2-353d-46c0-bb1b-097b8704be07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:14.407001 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-d02e5ec2-353d-46c0-bb1b-097b8704be07 None None] MFA Rules not processed for user `a293bc416349457ea83527a1ef42511f`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:14.408239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d02e5ec2-353d-46c0-bb1b-097b8704be07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:14.472943 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1223/2433] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:14 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:14.485098 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61c13ab-916a-4c7b-bd97-7c3673b5dc37 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:14.487031 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-b61c13ab-916a-4c7b-bd97-7c3673b5dc37 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:14.559989 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61c13ab-916a-4c7b-bd97-7c3673b5dc37 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:14.565502 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824ac07f-11f7-48c0-88ef-33a7b82cf664 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:14.566206 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824ac07f-11f7-48c0-88ef-33a7b82cf664 service nova] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:14.566490 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824ac07f-11f7-48c0-88ef-33a7b82cf664 service nova] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:14.566765 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824ac07f-11f7-48c0-88ef-33a7b82cf664 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:14.574863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824ac07f-11f7-48c0-88ef-33a7b82cf664 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:14.576430 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824ac07f-11f7-48c0-88ef-33a7b82cf664 service nova] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:14.585053 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1211/2434] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:26:14 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:26:14.788829 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02e5ec2-353d-46c0-bb1b-097b8704be07 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:14.848996 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02e5ec2-353d-46c0-bb1b-097b8704be07 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:14.853641 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e8ec7c-9181-43b9-858e-3565aabb35b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:14.854297 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e8ec7c-9181-43b9-858e-3565aabb35b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:14.854573 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e8ec7c-9181-43b9-858e-3565aabb35b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:14.854844 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e8ec7c-9181-43b9-858e-3565aabb35b4 admin admin] PATH_INFO: `/v3/users/a293bc416349457ea83527a1ef42511f` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:14.855973 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e8ec7c-9181-43b9-858e-3565aabb35b4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a293bc416349457ea83527a1ef42511f)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:14.857955 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e8ec7c-9181-43b9-858e-3565aabb35b4 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:14.887774 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:14.906712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-d8e8ec7c-9181-43b9-858e-3565aabb35b4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a293bc416349457ea83527a1ef42511f'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:14.914142 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-d8e8ec7c-9181-43b9-858e-3565aabb35b4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a293bc416349457ea83527a1ef42511f'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:14.915013 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-d8e8ec7c-9181-43b9-858e-3565aabb35b4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a293bc416349457ea83527a1ef42511f'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:14.921473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-d8e8ec7c-9181-43b9-858e-3565aabb35b4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a293bc416349457ea83527a1ef42511f'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:14.933834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-d8e8ec7c-9181-43b9-858e-3565aabb35b4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a293bc416349457ea83527a1ef42511f'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:14.939678 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1224/2435] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:26:14 2022] DELETE /identity/v3/users/a293bc416349457ea83527a1ef42511f => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:14.944153 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a295f6-0253-4347-8cca-b2d9f96880b1 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:14.944437 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a295f6-0253-4347-8cca-b2d9f96880b1 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:14.944712 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a295f6-0253-4347-8cca-b2d9f96880b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:14.963707 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-12a295f6-0253-4347-8cca-b2d9f96880b1 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:14.964898 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-12a295f6-0253-4347-8cca-b2d9f96880b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:14.993110 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=91,keystone:DELETE=6,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:15.026111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1212/2436] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:15.036082 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e8ec7c-9181-43b9-858e-3565aabb35b4 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:15.037456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d8e8ec7c-9181-43b9-858e-3565aabb35b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:15.099694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e8ec7c-9181-43b9-858e-3565aabb35b4 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:15.103983 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfaab63b-b58e-4fe3-a949-222d167b223c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:15.104582 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfaab63b-b58e-4fe3-a949-222d167b223c service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:15.104853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfaab63b-b58e-4fe3-a949-222d167b223c service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:15.105147 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfaab63b-b58e-4fe3-a949-222d167b223c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:15.114401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfaab63b-b58e-4fe3-a949-222d167b223c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:15.115683 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfaab63b-b58e-4fe3-a949-222d167b223c service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:15.125581 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1225/2437] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:26:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:26:15.433365 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a295f6-0253-4347-8cca-b2d9f96880b1 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:15.434827 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-12a295f6-0253-4347-8cca-b2d9f96880b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:15.450159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a295f6-0253-4347-8cca-b2d9f96880b1 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:15.455210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa39a7a6-a1c9-448d-9732-49e712c8cd1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:15.456050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa39a7a6-a1c9-448d-9732-49e712c8cd1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:15.456334 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa39a7a6-a1c9-448d-9732-49e712c8cd1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:15.456661 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa39a7a6-a1c9-448d-9732-49e712c8cd1a admin admin] PATH_INFO: `/v3/projects/49ef5f872e954a9295f1d2e1a00632b8` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:15.457786 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa39a7a6-a1c9-448d-9732-49e712c8cd1a admin admin] RBAC: Authorizing `identity:delete_project(project_id=49ef5f872e954a9295f1d2e1a00632b8)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:15.459662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa39a7a6-a1c9-448d-9732-49e712c8cd1a admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:15.514220 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-fa39a7a6-a1c9-448d-9732-49e712c8cd1a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '49ef5f872e954a9295f1d2e1a00632b8', 'request_id': 'req-fa39a7a6-a1c9-448d-9732-49e712c8cd1a'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:15.523075 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-fa39a7a6-a1c9-448d-9732-49e712c8cd1a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '49ef5f872e954a9295f1d2e1a00632b8', 'request_id': 'req-fa39a7a6-a1c9-448d-9732-49e712c8cd1a'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:15.530237 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-fa39a7a6-a1c9-448d-9732-49e712c8cd1a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '49ef5f872e954a9295f1d2e1a00632b8', 'request_id': 'req-fa39a7a6-a1c9-448d-9732-49e712c8cd1a'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:15.544103 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-fa39a7a6-a1c9-448d-9732-49e712c8cd1a admin admin] The token cache is being invalidate because project 49ef5f872e954a9295f1d2e1a00632b8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:26:15.544518 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-fa39a7a6-a1c9-448d-9732-49e712c8cd1a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:15.545293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-fa39a7a6-a1c9-448d-9732-49e712c8cd1a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:15.550573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1213/2438] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:26:15 2022] DELETE /identity/v3/projects/49ef5f872e954a9295f1d2e1a00632b8 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:15.558647 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931dd7d0-e5e5-41ed-b27a-ffb597830925 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:15.559688 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931dd7d0-e5e5-41ed-b27a-ffb597830925 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:15.560069 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931dd7d0-e5e5-41ed-b27a-ffb597830925 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:15.599932 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-931dd7d0-e5e5-41ed-b27a-ffb597830925 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:15.601223 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-931dd7d0-e5e5-41ed-b27a-ffb597830925 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:15.731555 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1226/2439] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:15.774655 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae977a8-85b0-429b-a879-fd92400c0007 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:15.774655 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae977a8-85b0-429b-a879-fd92400c0007 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:15.774655 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae977a8-85b0-429b-a879-fd92400c0007 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:15.774655 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-7ae977a8-85b0-429b-a879-fd92400c0007 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:15.774655 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-7ae977a8-85b0-429b-a879-fd92400c0007 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:15.774655 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1214/2440] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:15.775828 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931dd7d0-e5e5-41ed-b27a-ffb597830925 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:15.791428 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931dd7d0-e5e5-41ed-b27a-ffb597830925 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:15.795996 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60428528-5d01-4d94-b077-4d1b140caf8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:15.803835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60428528-5d01-4d94-b077-4d1b140caf8b admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:15.803835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60428528-5d01-4d94-b077-4d1b140caf8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:15.803835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60428528-5d01-4d94-b077-4d1b140caf8b admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:15.803835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60428528-5d01-4d94-b077-4d1b140caf8b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:26:15.803835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60428528-5d01-4d94-b077-4d1b140caf8b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:15.803835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60428528-5d01-4d94-b077-4d1b140caf8b admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:15.817016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1227/2441] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:26:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:26:15.821925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae977a8-85b0-429b-a879-fd92400c0007 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:15.831298 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae977a8-85b0-429b-a879-fd92400c0007 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:15.835838 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042d0dbf-8060-4ae6-90b7-bf10f5298fb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:15.836427 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042d0dbf-8060-4ae6-90b7-bf10f5298fb0 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:15.836704 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042d0dbf-8060-4ae6-90b7-bf10f5298fb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:15.836974 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042d0dbf-8060-4ae6-90b7-bf10f5298fb0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:15.837622 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042d0dbf-8060-4ae6-90b7-bf10f5298fb0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:15.839580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042d0dbf-8060-4ae6-90b7-bf10f5298fb0 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:15.866114 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1215/2442] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:26:15 2022] POST /identity/v3/projects => generated 403 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:26:15.870929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60428528-5d01-4d94-b077-4d1b140caf8b admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:15.880764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60428528-5d01-4d94-b077-4d1b140caf8b admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:15.911867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:15.938992 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dccf7402-8494-45f6-9986-ced878558da2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:15.939582 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dccf7402-8494-45f6-9986-ced878558da2 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:15.939858 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dccf7402-8494-45f6-9986-ced878558da2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:15.940131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dccf7402-8494-45f6-9986-ced878558da2 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:15.940740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dccf7402-8494-45f6-9986-ced878558da2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:15.942811 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dccf7402-8494-45f6-9986-ced878558da2 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:15.963556 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1228/2443] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:26:15 2022] POST /identity/v3/users => generated 302 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:26:15.967619 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042d0dbf-8060-4ae6-90b7-bf10f5298fb0 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:15.977845 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042d0dbf-8060-4ae6-90b7-bf10f5298fb0 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:15.982912 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e310ef-0c1d-4ca7-a6f6-be92885c1eae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:15.983503 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e310ef-0c1d-4ca7-a6f6-be92885c1eae admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:15.983818 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e310ef-0c1d-4ca7-a6f6-be92885c1eae admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:15.984092 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e310ef-0c1d-4ca7-a6f6-be92885c1eae admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:15.984762 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e310ef-0c1d-4ca7-a6f6-be92885c1eae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:15.986387 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e310ef-0c1d-4ca7-a6f6-be92885c1eae admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:15.996677 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1216/2444] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:15 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:16.000504 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dccf7402-8494-45f6-9986-ced878558da2 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:16.010715 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dccf7402-8494-45f6-9986-ced878558da2 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:16.015517 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a84acb-a2d8-4ec7-92a8-478f896a843d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:16.016112 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a84acb-a2d8-4ec7-92a8-478f896a843d admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:16.016391 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a84acb-a2d8-4ec7-92a8-478f896a843d admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:16.016666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a84acb-a2d8-4ec7-92a8-478f896a843d admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:16.017304 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a84acb-a2d8-4ec7-92a8-478f896a843d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:16.018990 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a84acb-a2d8-4ec7-92a8-478f896a843d admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:16.029534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1229/2445] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:15 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:16.033984 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e310ef-0c1d-4ca7-a6f6-be92885c1eae admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:16.045516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e310ef-0c1d-4ca7-a6f6-be92885c1eae admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:16.050189 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed5c257-8834-4bff-a843-f6acbd86471e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:16.050833 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed5c257-8834-4bff-a843-f6acbd86471e admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:16.051115 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed5c257-8834-4bff-a843-f6acbd86471e admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:16.051391 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed5c257-8834-4bff-a843-f6acbd86471e admin admin] PATH_INFO: `/v3/projects/4a3fd16699244bbd88bd49792d3972c1/users/1c448347ce07446f842fc6c8263fa484/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:16.066497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed5c257-8834-4bff-a843-f6acbd86471e admin admin] RBAC: Authorizing `identity:create_grant(project_id=4a3fd16699244bbd88bd49792d3972c1, user_id=1c448347ce07446f842fc6c8263fa484, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:16.069118 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed5c257-8834-4bff-a843-f6acbd86471e admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:16.081724 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1217/2446] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:26:16 2022] PUT /identity/v3/projects/4a3fd16699244bbd88bd49792d3972c1/users/1c448347ce07446f842fc6c8263fa484/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:26:16.086745 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e770dec3-38a5-40c5-9064-9e7d8e231b24 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:16.087035 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e770dec3-38a5-40c5-9064-9e7d8e231b24 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:16.087314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e770dec3-38a5-40c5-9064-9e7d8e231b24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:16.107011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-e770dec3-38a5-40c5-9064-9e7d8e231b24 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:16.108201 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-e770dec3-38a5-40c5-9064-9e7d8e231b24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:16.166477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1230/2447] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:16.176031 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed5c257-8834-4bff-a843-f6acbd86471e admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:16.177685 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-fed5c257-8834-4bff-a843-f6acbd86471e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:16.234206 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed5c257-8834-4bff-a843-f6acbd86471e admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:16.237811 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fd14a7-e999-4dd3-bccf-4cdfd722a258 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:16.238408 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fd14a7-e999-4dd3-bccf-4cdfd722a258 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:16.238685 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fd14a7-e999-4dd3-bccf-4cdfd722a258 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:16.238957 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fd14a7-e999-4dd3-bccf-4cdfd722a258 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:16.245750 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:16.247967 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fd14a7-e999-4dd3-bccf-4cdfd722a258 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:16.249862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fd14a7-e999-4dd3-bccf-4cdfd722a258 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:16.260606 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1218/2448] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:26:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:26:17.212658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3231840-7c18-45a6-80a2-07a1ef844178 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:17.213824 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3231840-7c18-45a6-80a2-07a1ef844178 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:17.214118 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3231840-7c18-45a6-80a2-07a1ef844178 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:17.243950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-f3231840-7c18-45a6-80a2-07a1ef844178 None None] MFA Rules not processed for user `1c448347ce07446f842fc6c8263fa484`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:17.245154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-f3231840-7c18-45a6-80a2-07a1ef844178 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:17.314930 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1231/2449] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:17 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:17.319494 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fd14a7-e999-4dd3-bccf-4cdfd722a258 service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:17.330303 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fd14a7-e999-4dd3-bccf-4cdfd722a258 service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:17.335166 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f16746-51b9-480b-8e03-b1ba8c73b437 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:17.336091 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f16746-51b9-480b-8e03-b1ba8c73b437 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:17.336629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f16746-51b9-480b-8e03-b1ba8c73b437 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:17.337212 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f16746-51b9-480b-8e03-b1ba8c73b437 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:17.338112 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f16746-51b9-480b-8e03-b1ba8c73b437 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:17.340486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f16746-51b9-480b-8e03-b1ba8c73b437 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:17.375100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1219/2450] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:26:17 2022] POST /identity/v3/projects => generated 403 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:26:17.376718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3231840-7c18-45a6-80a2-07a1ef844178 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:17.388976 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3231840-7c18-45a6-80a2-07a1ef844178 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:17.443685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:17.466380 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9088e04-4354-4dbd-9261-e102ee8f6756 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:17.467683 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9088e04-4354-4dbd-9261-e102ee8f6756 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:17.467961 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9088e04-4354-4dbd-9261-e102ee8f6756 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:17.468457 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9088e04-4354-4dbd-9261-e102ee8f6756 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:17.469256 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9088e04-4354-4dbd-9261-e102ee8f6756 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:17.471234 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9088e04-4354-4dbd-9261-e102ee8f6756 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:17.497904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1232/2451] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:26:17 2022] POST /identity/v3/users => generated 308 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:26:17.502497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f16746-51b9-480b-8e03-b1ba8c73b437 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:17.519447 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f16746-51b9-480b-8e03-b1ba8c73b437 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:17.530090 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91661e1c-56c3-4e8b-91a5-b2b59eaa5b04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:17.531199 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91661e1c-56c3-4e8b-91a5-b2b59eaa5b04 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:17.532589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91661e1c-56c3-4e8b-91a5-b2b59eaa5b04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:17.534053 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91661e1c-56c3-4e8b-91a5-b2b59eaa5b04 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:17.535075 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91661e1c-56c3-4e8b-91a5-b2b59eaa5b04 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:17.537391 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91661e1c-56c3-4e8b-91a5-b2b59eaa5b04 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:17.566943 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1220/2452] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:17 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:17.571449 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9088e04-4354-4dbd-9261-e102ee8f6756 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:17.582614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9088e04-4354-4dbd-9261-e102ee8f6756 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:17.588912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b42da9b-ff43-491e-8358-ebbacc7626fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:17.589692 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b42da9b-ff43-491e-8358-ebbacc7626fc admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:17.590289 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b42da9b-ff43-491e-8358-ebbacc7626fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:17.590435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b42da9b-ff43-491e-8358-ebbacc7626fc admin admin] PATH_INFO: `/v3/projects/79e01f607d3545c9a76cd6ab096aeeac/users/889f1a1951934afbbf3d185b7023c676/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:17.615030 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b42da9b-ff43-491e-8358-ebbacc7626fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=79e01f607d3545c9a76cd6ab096aeeac, user_id=889f1a1951934afbbf3d185b7023c676, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:17.619804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b42da9b-ff43-491e-8358-ebbacc7626fc admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:17.636985 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1233/2453] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:26:17 2022] PUT /identity/v3/projects/79e01f607d3545c9a76cd6ab096aeeac/users/889f1a1951934afbbf3d185b7023c676/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:17.641814 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91661e1c-56c3-4e8b-91a5-b2b59eaa5b04 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:17.652380 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91661e1c-56c3-4e8b-91a5-b2b59eaa5b04 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:17.719398 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2b2414-5667-4471-9c83-96a4ce015a8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:17.720019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2b2414-5667-4471-9c83-96a4ce015a8c admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:17.720302 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2b2414-5667-4471-9c83-96a4ce015a8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:17.720576 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2b2414-5667-4471-9c83-96a4ce015a8c admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:17.721228 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2b2414-5667-4471-9c83-96a4ce015a8c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:17.722918 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2b2414-5667-4471-9c83-96a4ce015a8c admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:17.740400 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1221/2454] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:17 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:26:17.744882 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b42da9b-ff43-491e-8358-ebbacc7626fc admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:17.760737 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b42da9b-ff43-491e-8358-ebbacc7626fc admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:17.769781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3045e044-6959-467a-9fdb-70a615b45bef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:17.770703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3045e044-6959-467a-9fdb-70a615b45bef admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:17.771139 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3045e044-6959-467a-9fdb-70a615b45bef admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:17.771423 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3045e044-6959-467a-9fdb-70a615b45bef admin admin] PATH_INFO: `/v3/projects/79e01f607d3545c9a76cd6ab096aeeac/users/889f1a1951934afbbf3d185b7023c676/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:17.773496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3045e044-6959-467a-9fdb-70a615b45bef admin admin] RBAC: Authorizing `identity:create_grant(project_id=79e01f607d3545c9a76cd6ab096aeeac, user_id=889f1a1951934afbbf3d185b7023c676, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:17.776919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3045e044-6959-467a-9fdb-70a615b45bef admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:17.788915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1234/2455] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:26:17 2022] PUT /identity/v3/projects/79e01f607d3545c9a76cd6ab096aeeac/users/889f1a1951934afbbf3d185b7023c676/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:18.452723 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b94006a-ed3a-44f4-9939-d75c1dc02ae4 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:18.453334 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b94006a-ed3a-44f4-9939-d75c1dc02ae4 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:18.453334 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b94006a-ed3a-44f4-9939-d75c1dc02ae4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:18.483635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-0b94006a-ed3a-44f4-9939-d75c1dc02ae4 None None] MFA Rules not processed for user `889f1a1951934afbbf3d185b7023c676`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:18.485098 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0b94006a-ed3a-44f4-9939-d75c1dc02ae4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:18.555422 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1222/2456] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:18 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:19.006363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e6c173-0acb-4e09-8aaa-4b08234e152a None None] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:19.006825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e6c173-0acb-4e09-8aaa-4b08234e152a None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:19.007214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e6c173-0acb-4e09-8aaa-4b08234e152a None None] PATH_INFO: `/` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:19.008930 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1235/2457] 10.222.0.26 () {54 vars in 961 bytes} [Tue Sep 27 04:26:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 04:26:19.015147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb394c69-79e8-4faf-a3c8-106745f22444 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:19.015591 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb394c69-79e8-4faf-a3c8-106745f22444 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:19.015976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb394c69-79e8-4faf-a3c8-106745f22444 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:19.045025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-bb394c69-79e8-4faf-a3c8-106745f22444 None None] MFA Rules not processed for user `3a1bc198dfc0486e9e56d4fb28cda05d`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:19.046437 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-bb394c69-79e8-4faf-a3c8-106745f22444 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:19.121861 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:19.153901 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1223/2458] 10.222.0.26 () {58 vars in 1058 bytes} [Tue Sep 27 04:26:19 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:26:19.160977 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e6c173-0acb-4e09-8aaa-4b08234e152a None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:19.177912 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e6c173-0acb-4e09-8aaa-4b08234e152a None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:19.182368 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f6c4d0-5e0c-480e-b3a6-7812ce81539e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3a1bc198dfc0486e9e56d4fb28cda05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:19.183044 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f6c4d0-5e0c-480e-b3a6-7812ce81539e service placement] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:19.183381 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f6c4d0-5e0c-480e-b3a6-7812ce81539e service placement] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:19.183712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f6c4d0-5e0c-480e-b3a6-7812ce81539e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:19.185533 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-80f6c4d0-5e0c-480e-b3a6-7812ce81539e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:19.198343 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f6c4d0-5e0c-480e-b3a6-7812ce81539e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:19.199995 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f6c4d0-5e0c-480e-b3a6-7812ce81539e service placement] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:19.209720 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1236/2459] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:26:19 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:26:19.487044 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf67616-bf22-4dd3-8c95-f293cd2228a4 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:19.487542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf67616-bf22-4dd3-8c95-f293cd2228a4 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:19.487634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf67616-bf22-4dd3-8c95-f293cd2228a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:19.507034 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-2cf67616-bf22-4dd3-8c95-f293cd2228a4 None None] MFA Rules not processed for user `5a2fcc8c9ac44dceadcc30d163bbfa0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:19.508678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-2cf67616-bf22-4dd3-8c95-f293cd2228a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:19.526150 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f6c4d0-5e0c-480e-b3a6-7812ce81539e service placement] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:19.539816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f6c4d0-5e0c-480e-b3a6-7812ce81539e service placement] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:19.547420 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780a0294-c52b-435a-9625-83ef1ace0030 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3a1bc198dfc0486e9e56d4fb28cda05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:19.548205 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780a0294-c52b-435a-9625-83ef1ace0030 service placement] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:19.548636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780a0294-c52b-435a-9625-83ef1ace0030 service placement] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:19.549060 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780a0294-c52b-435a-9625-83ef1ace0030 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:19.551651 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-780a0294-c52b-435a-9625-83ef1ace0030 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:19.565919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-780a0294-c52b-435a-9625-83ef1ace0030 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:19.567681 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-780a0294-c52b-435a-9625-83ef1ace0030 service placement] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:19.572580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1224/2460] 10.222.0.26 () {58 vars in 967 bytes} [Tue Sep 27 04:26:19 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:19.580427 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1237/2461] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:26:19 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:26:19.588005 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf67616-bf22-4dd3-8c95-f293cd2228a4 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:19.589798 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-2cf67616-bf22-4dd3-8c95-f293cd2228a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:19.595732 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780a0294-c52b-435a-9625-83ef1ace0030 service placement] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:19.668689 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf67616-bf22-4dd3-8c95-f293cd2228a4 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:19.669172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780a0294-c52b-435a-9625-83ef1ace0030 service placement] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:19.673645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9185417e-063e-493f-9e06-83ad7d8ae2c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:19.673837 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4247560f-137d-46f9-b74d-9d66833273a1 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '181dd5dc720c4a97860852062b7b19c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:19.674288 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4247560f-137d-46f9-b74d-9d66833273a1 service swift] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:19.674397 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9185417e-063e-493f-9e06-83ad7d8ae2c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:19.674564 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4247560f-137d-46f9-b74d-9d66833273a1 service swift] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:19.674682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9185417e-063e-493f-9e06-83ad7d8ae2c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:19.674834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4247560f-137d-46f9-b74d-9d66833273a1 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:19.674944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9185417e-063e-493f-9e06-83ad7d8ae2c5 admin admin] PATH_INFO: `/v3/users/1c448347ce07446f842fc6c8263fa484` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:19.675987 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9185417e-063e-493f-9e06-83ad7d8ae2c5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1c448347ce07446f842fc6c8263fa484)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:19.677945 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9185417e-063e-493f-9e06-83ad7d8ae2c5 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:19.683548 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4247560f-137d-46f9-b74d-9d66833273a1 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:19.685099 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4247560f-137d-46f9-b74d-9d66833273a1 service swift] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:19.694118 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1225/2462] 10.222.0.26 () {60 vars in 1384 bytes} [Tue Sep 27 04:26:19 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 04:26:19.743583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-9185417e-063e-493f-9e06-83ad7d8ae2c5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1c448347ce07446f842fc6c8263fa484'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:19.751393 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-9185417e-063e-493f-9e06-83ad7d8ae2c5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1c448347ce07446f842fc6c8263fa484'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:19.752072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-9185417e-063e-493f-9e06-83ad7d8ae2c5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1c448347ce07446f842fc6c8263fa484'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:19.760340 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-9185417e-063e-493f-9e06-83ad7d8ae2c5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1c448347ce07446f842fc6c8263fa484'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:19.768124 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=85,keystone:INSERT=6,keystone:DELETE=9 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:19.774084 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-9185417e-063e-493f-9e06-83ad7d8ae2c5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1c448347ce07446f842fc6c8263fa484'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:19.782094 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1238/2463] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:26:19 2022] DELETE /identity/v3/users/1c448347ce07446f842fc6c8263fa484 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:26:19.962101 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4247560f-137d-46f9-b74d-9d66833273a1 service swift] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:19.980901 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4247560f-137d-46f9-b74d-9d66833273a1 service swift] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:20.043068 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b370fcb-39a4-4ba6-af09-e9e5b6996803 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:20.043719 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b370fcb-39a4-4ba6-af09-e9e5b6996803 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:20.044001 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b370fcb-39a4-4ba6-af09-e9e5b6996803 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:20.044271 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b370fcb-39a4-4ba6-af09-e9e5b6996803 admin admin] PATH_INFO: `/v3/projects/4a3fd16699244bbd88bd49792d3972c1` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:20.045380 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b370fcb-39a4-4ba6-af09-e9e5b6996803 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4a3fd16699244bbd88bd49792d3972c1)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:20.047296 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b370fcb-39a4-4ba6-af09-e9e5b6996803 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:20.064750 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=100 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:20.097719 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-4b370fcb-39a4-4ba6-af09-e9e5b6996803 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4a3fd16699244bbd88bd49792d3972c1', 'request_id': 'req-4b370fcb-39a4-4ba6-af09-e9e5b6996803'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:20.104309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-4b370fcb-39a4-4ba6-af09-e9e5b6996803 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4a3fd16699244bbd88bd49792d3972c1', 'request_id': 'req-4b370fcb-39a4-4ba6-af09-e9e5b6996803'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:20.112518 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-4b370fcb-39a4-4ba6-af09-e9e5b6996803 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4a3fd16699244bbd88bd49792d3972c1', 'request_id': 'req-4b370fcb-39a4-4ba6-af09-e9e5b6996803'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:20.126987 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-4b370fcb-39a4-4ba6-af09-e9e5b6996803 admin admin] The token cache is being invalidate because project 4a3fd16699244bbd88bd49792d3972c1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:26:20.127283 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-4b370fcb-39a4-4ba6-af09-e9e5b6996803 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:20.128065 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-4b370fcb-39a4-4ba6-af09-e9e5b6996803 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:20.129773 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1226/2464] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:26:19 2022] DELETE /identity/v3/projects/4a3fd16699244bbd88bd49792d3972c1 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:20.133615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9185417e-063e-493f-9e06-83ad7d8ae2c5 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:20.135593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-9185417e-063e-493f-9e06-83ad7d8ae2c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:20.148971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9185417e-063e-493f-9e06-83ad7d8ae2c5 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:20.242134 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608ec211-8ab8-45ae-9435-ec6523ab6510 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:20.243015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608ec211-8ab8-45ae-9435-ec6523ab6510 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:20.243391 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608ec211-8ab8-45ae-9435-ec6523ab6510 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:20.243761 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608ec211-8ab8-45ae-9435-ec6523ab6510 admin admin] PATH_INFO: `/v3/users/889f1a1951934afbbf3d185b7023c676` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:20.245608 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608ec211-8ab8-45ae-9435-ec6523ab6510 admin admin] RBAC: Authorizing `identity:delete_user(user_id=889f1a1951934afbbf3d185b7023c676)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:20.247819 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608ec211-8ab8-45ae-9435-ec6523ab6510 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:20.306703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-608ec211-8ab8-45ae-9435-ec6523ab6510 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '889f1a1951934afbbf3d185b7023c676'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:20.313695 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-608ec211-8ab8-45ae-9435-ec6523ab6510 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '889f1a1951934afbbf3d185b7023c676'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:20.314431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-608ec211-8ab8-45ae-9435-ec6523ab6510 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '889f1a1951934afbbf3d185b7023c676'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:20.322571 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-608ec211-8ab8-45ae-9435-ec6523ab6510 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '889f1a1951934afbbf3d185b7023c676'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:20.334618 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-608ec211-8ab8-45ae-9435-ec6523ab6510 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '889f1a1951934afbbf3d185b7023c676'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:20.341057 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1239/2465] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:26:20 2022] DELETE /identity/v3/users/889f1a1951934afbbf3d185b7023c676 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:20.517316 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b370fcb-39a4-4ba6-af09-e9e5b6996803 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:20.518874 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-4b370fcb-39a4-4ba6-af09-e9e5b6996803 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:20.533962 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b370fcb-39a4-4ba6-af09-e9e5b6996803 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:20.597732 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f1ba5b-665e-4546-8fe7-a11bb2c91a62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:20.598392 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f1ba5b-665e-4546-8fe7-a11bb2c91a62 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:20.598671 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f1ba5b-665e-4546-8fe7-a11bb2c91a62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:20.598945 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f1ba5b-665e-4546-8fe7-a11bb2c91a62 admin admin] PATH_INFO: `/v3/projects/79e01f607d3545c9a76cd6ab096aeeac` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:20.600049 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f1ba5b-665e-4546-8fe7-a11bb2c91a62 admin admin] RBAC: Authorizing `identity:delete_project(project_id=79e01f607d3545c9a76cd6ab096aeeac)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:20.601967 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f1ba5b-665e-4546-8fe7-a11bb2c91a62 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:20.647611 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-03f1ba5b-665e-4546-8fe7-a11bb2c91a62 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '79e01f607d3545c9a76cd6ab096aeeac', 'request_id': 'req-03f1ba5b-665e-4546-8fe7-a11bb2c91a62'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:20.653104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-03f1ba5b-665e-4546-8fe7-a11bb2c91a62 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '79e01f607d3545c9a76cd6ab096aeeac', 'request_id': 'req-03f1ba5b-665e-4546-8fe7-a11bb2c91a62'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:20.659203 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-03f1ba5b-665e-4546-8fe7-a11bb2c91a62 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '79e01f607d3545c9a76cd6ab096aeeac', 'request_id': 'req-03f1ba5b-665e-4546-8fe7-a11bb2c91a62'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:20.671631 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-03f1ba5b-665e-4546-8fe7-a11bb2c91a62 admin admin] The token cache is being invalidate because project 79e01f607d3545c9a76cd6ab096aeeac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:26:20.671923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-03f1ba5b-665e-4546-8fe7-a11bb2c91a62 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:20.672553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-03f1ba5b-665e-4546-8fe7-a11bb2c91a62 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:20.674094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1227/2466] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:26:20 2022] DELETE /identity/v3/projects/79e01f607d3545c9a76cd6ab096aeeac => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:20.679145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f71ec38-b6af-4221-b0cf-06945e9e1d01 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:20.679423 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f71ec38-b6af-4221-b0cf-06945e9e1d01 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:20.679698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f71ec38-b6af-4221-b0cf-06945e9e1d01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:20.699792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-5f71ec38-b6af-4221-b0cf-06945e9e1d01 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:20.700974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-5f71ec38-b6af-4221-b0cf-06945e9e1d01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:20.758581 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1240/2467] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:20.767093 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6acce22-1ff8-43f5-823e-3b916f0f6c4d None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:20.767322 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6acce22-1ff8-43f5-823e-3b916f0f6c4d None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:20.767601 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6acce22-1ff8-43f5-823e-3b916f0f6c4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:20.786842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-d6acce22-1ff8-43f5-823e-3b916f0f6c4d None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:20.788283 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-d6acce22-1ff8-43f5-823e-3b916f0f6c4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:20.796076 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1228/2468] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:20.800111 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f71ec38-b6af-4221-b0cf-06945e9e1d01 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:20.810132 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f71ec38-b6af-4221-b0cf-06945e9e1d01 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:20.814568 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b134ccb-7414-4205-9390-d5d903344bf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:20.815152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b134ccb-7414-4205-9390-d5d903344bf5 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:20.815428 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b134ccb-7414-4205-9390-d5d903344bf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:20.815696 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b134ccb-7414-4205-9390-d5d903344bf5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:20.816333 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b134ccb-7414-4205-9390-d5d903344bf5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:26:20.816629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b134ccb-7414-4205-9390-d5d903344bf5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:20.818224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b134ccb-7414-4205-9390-d5d903344bf5 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:20.820686 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:20.831645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1241/2469] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:26:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:26:20.835688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6acce22-1ff8-43f5-823e-3b916f0f6c4d None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:20.845151 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6acce22-1ff8-43f5-823e-3b916f0f6c4d None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:20.849868 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b26939b-e4d3-4bb9-821b-106bc260af1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:20.850459 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b26939b-e4d3-4bb9-821b-106bc260af1a admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:20.850734 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b26939b-e4d3-4bb9-821b-106bc260af1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:20.851001 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b26939b-e4d3-4bb9-821b-106bc260af1a admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:20.851630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b26939b-e4d3-4bb9-821b-106bc260af1a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:20.853527 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b26939b-e4d3-4bb9-821b-106bc260af1a admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:20.880427 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1229/2470] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:26:20 2022] POST /identity/v3/projects => generated 375 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:26:20.885007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b134ccb-7414-4205-9390-d5d903344bf5 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:20.894836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b134ccb-7414-4205-9390-d5d903344bf5 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:20.949432 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10514d64-91f6-49a0-8097-ce76a955e1d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:20.950046 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10514d64-91f6-49a0-8097-ce76a955e1d6 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:20.950321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10514d64-91f6-49a0-8097-ce76a955e1d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:20.950591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10514d64-91f6-49a0-8097-ce76a955e1d6 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:20.951316 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10514d64-91f6-49a0-8097-ce76a955e1d6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:20.953242 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10514d64-91f6-49a0-8097-ce76a955e1d6 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:20.973197 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1242/2471] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:26:20 2022] POST /identity/v3/users => generated 288 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:26:20.977328 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b26939b-e4d3-4bb9-821b-106bc260af1a admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:20.987833 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b26939b-e4d3-4bb9-821b-106bc260af1a admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:20.992323 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742fdc1c-8712-46d3-96dc-8c4cbe4a344c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:20.992907 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742fdc1c-8712-46d3-96dc-8c4cbe4a344c admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:20.993184 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742fdc1c-8712-46d3-96dc-8c4cbe4a344c admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:20.993451 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742fdc1c-8712-46d3-96dc-8c4cbe4a344c admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:20.994802 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742fdc1c-8712-46d3-96dc-8c4cbe4a344c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:20.996374 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742fdc1c-8712-46d3-96dc-8c4cbe4a344c admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:20.997648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:21.007525 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1230/2472] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:21.011362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10514d64-91f6-49a0-8097-ce76a955e1d6 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:21.021473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10514d64-91f6-49a0-8097-ce76a955e1d6 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:21.026033 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec63c257-5410-482c-bd8a-487a53031222 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:21.026627 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec63c257-5410-482c-bd8a-487a53031222 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:21.026907 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec63c257-5410-482c-bd8a-487a53031222 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:21.027175 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec63c257-5410-482c-bd8a-487a53031222 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:21.027810 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec63c257-5410-482c-bd8a-487a53031222 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:21.029418 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec63c257-5410-482c-bd8a-487a53031222 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:21.040056 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1243/2473] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:21 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:21.044626 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742fdc1c-8712-46d3-96dc-8c4cbe4a344c admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:21.054274 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742fdc1c-8712-46d3-96dc-8c4cbe4a344c admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:21.059439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48dbadd6-84ef-4bc1-acee-9ce3abe1c865 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:21.060070 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dbadd6-84ef-4bc1-acee-9ce3abe1c865 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:21.060346 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dbadd6-84ef-4bc1-acee-9ce3abe1c865 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:21.060616 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dbadd6-84ef-4bc1-acee-9ce3abe1c865 admin admin] PATH_INFO: `/v3/projects/80ffbd2c0065498eabe1a93e2e3e0111/users/36c8cb515abf41b291ea187c8d0660cf/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:21.075167 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48dbadd6-84ef-4bc1-acee-9ce3abe1c865 admin admin] RBAC: Authorizing `identity:create_grant(project_id=80ffbd2c0065498eabe1a93e2e3e0111, user_id=36c8cb515abf41b291ea187c8d0660cf, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:21.077881 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48dbadd6-84ef-4bc1-acee-9ce3abe1c865 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:21.091250 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1231/2474] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:26:21 2022] PUT /identity/v3/projects/80ffbd2c0065498eabe1a93e2e3e0111/users/36c8cb515abf41b291ea187c8d0660cf/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:21.101302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5315063f-4185-4a21-82ed-9d411fa5221d None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:21.101583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5315063f-4185-4a21-82ed-9d411fa5221d None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:21.101889 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5315063f-4185-4a21-82ed-9d411fa5221d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:21.130156 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-5315063f-4185-4a21-82ed-9d411fa5221d None None] MFA Rules not processed for user `36c8cb515abf41b291ea187c8d0660cf`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:21.131341 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-5315063f-4185-4a21-82ed-9d411fa5221d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:21.195150 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1244/2475] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:21 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:21.199667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48dbadd6-84ef-4bc1-acee-9ce3abe1c865 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:21.261462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48dbadd6-84ef-4bc1-acee-9ce3abe1c865 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:21.266177 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb43d33-193f-425f-afd3-7d02b9c4795a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:21.266706 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb43d33-193f-425f-afd3-7d02b9c4795a admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:21.266987 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb43d33-193f-425f-afd3-7d02b9c4795a admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:21.267255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb43d33-193f-425f-afd3-7d02b9c4795a admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:21.267884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb43d33-193f-425f-afd3-7d02b9c4795a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:21.269760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb43d33-193f-425f-afd3-7d02b9c4795a admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:21.299612 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1232/2476] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:26:21 2022] POST /identity/v3/projects => generated 377 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:26:21.303687 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5315063f-4185-4a21-82ed-9d411fa5221d None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:21.321181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5315063f-4185-4a21-82ed-9d411fa5221d None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:21.378134 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c74804-7362-43b0-a5ba-5401b62fee1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:21.378742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c74804-7362-43b0-a5ba-5401b62fee1f admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:21.379015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c74804-7362-43b0-a5ba-5401b62fee1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:21.379283 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c74804-7362-43b0-a5ba-5401b62fee1f admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:21.379903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c74804-7362-43b0-a5ba-5401b62fee1f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:21.381806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c74804-7362-43b0-a5ba-5401b62fee1f admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:21.391817 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:21.410457 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1245/2477] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:26:21 2022] POST /identity/v3/users => generated 295 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:26:21.414475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb43d33-193f-425f-afd3-7d02b9c4795a admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:21.426171 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb43d33-193f-425f-afd3-7d02b9c4795a admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:21.430912 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f524f62-9a81-4e26-b43d-0e46ce4c27db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:21.431620 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f524f62-9a81-4e26-b43d-0e46ce4c27db admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:21.431927 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f524f62-9a81-4e26-b43d-0e46ce4c27db admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:21.432206 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f524f62-9a81-4e26-b43d-0e46ce4c27db admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:21.433172 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f524f62-9a81-4e26-b43d-0e46ce4c27db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:21.434891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f524f62-9a81-4e26-b43d-0e46ce4c27db admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:21.445583 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1233/2478] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:21 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:26:21.449926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c74804-7362-43b0-a5ba-5401b62fee1f admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:21.459692 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c74804-7362-43b0-a5ba-5401b62fee1f admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:21.464745 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0485b23e-1b39-44c4-9544-728b823f2c87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:21.465381 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0485b23e-1b39-44c4-9544-728b823f2c87 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:21.465655 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0485b23e-1b39-44c4-9544-728b823f2c87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:21.465958 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0485b23e-1b39-44c4-9544-728b823f2c87 admin admin] PATH_INFO: `/v3/projects/6ebc46fe189d440599fb32a60fa430c2/users/b9feb2405c7b4194a87b075f16fcac6c/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:21.480888 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0485b23e-1b39-44c4-9544-728b823f2c87 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6ebc46fe189d440599fb32a60fa430c2, user_id=b9feb2405c7b4194a87b075f16fcac6c, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:21.483534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0485b23e-1b39-44c4-9544-728b823f2c87 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:21.495880 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1246/2479] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:26:21 2022] PUT /identity/v3/projects/6ebc46fe189d440599fb32a60fa430c2/users/b9feb2405c7b4194a87b075f16fcac6c/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:21.500015 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f524f62-9a81-4e26-b43d-0e46ce4c27db admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:21.510252 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f524f62-9a81-4e26-b43d-0e46ce4c27db admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:21.565706 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416af8dc-34d9-426c-8753-57511f453a9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:21.566357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416af8dc-34d9-426c-8753-57511f453a9c admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:21.566537 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416af8dc-34d9-426c-8753-57511f453a9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:21.566811 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416af8dc-34d9-426c-8753-57511f453a9c admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:21.567448 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416af8dc-34d9-426c-8753-57511f453a9c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:21.569026 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416af8dc-34d9-426c-8753-57511f453a9c admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:21.578674 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1234/2480] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:21 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:21.582917 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0485b23e-1b39-44c4-9544-728b823f2c87 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:21.593392 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0485b23e-1b39-44c4-9544-728b823f2c87 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:21.597926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f24c4ee-bf16-49e2-b3a2-3d286483df6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:21.598567 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f24c4ee-bf16-49e2-b3a2-3d286483df6e admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:21.598836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f24c4ee-bf16-49e2-b3a2-3d286483df6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:21.599103 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f24c4ee-bf16-49e2-b3a2-3d286483df6e admin admin] PATH_INFO: `/v3/projects/6ebc46fe189d440599fb32a60fa430c2/users/b9feb2405c7b4194a87b075f16fcac6c/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:21.600952 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f24c4ee-bf16-49e2-b3a2-3d286483df6e admin admin] RBAC: Authorizing `identity:create_grant(project_id=6ebc46fe189d440599fb32a60fa430c2, user_id=b9feb2405c7b4194a87b075f16fcac6c, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:21.604204 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f24c4ee-bf16-49e2-b3a2-3d286483df6e admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:21.614069 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1247/2481] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:26:21 2022] PUT /identity/v3/projects/6ebc46fe189d440599fb32a60fa430c2/users/b9feb2405c7b4194a87b075f16fcac6c/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:21.624976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b40a677-dd10-41fc-bf25-1eb649faa71c None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:21.625364 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b40a677-dd10-41fc-bf25-1eb649faa71c None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:21.625699 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b40a677-dd10-41fc-bf25-1eb649faa71c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:21.653413 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-0b40a677-dd10-41fc-bf25-1eb649faa71c None None] MFA Rules not processed for user `b9feb2405c7b4194a87b075f16fcac6c`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:21.654632 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0b40a677-dd10-41fc-bf25-1eb649faa71c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:21.684382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:21.726251 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1235/2482] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:21 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:21.737017 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f24c4ee-bf16-49e2-b3a2-3d286483df6e admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:21.738775 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-5f24c4ee-bf16-49e2-b3a2-3d286483df6e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:21.795803 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f24c4ee-bf16-49e2-b3a2-3d286483df6e admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:21.799399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef8de03-347b-4dec-be69-2307179bca92 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:21.799993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef8de03-347b-4dec-be69-2307179bca92 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:21.800272 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef8de03-347b-4dec-be69-2307179bca92 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:21.800540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef8de03-347b-4dec-be69-2307179bca92 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:21.856688 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef8de03-347b-4dec-be69-2307179bca92 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:21.858080 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef8de03-347b-4dec-be69-2307179bca92 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:21.866836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1248/2483] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:26:21 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:26:26.472275 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b40a677-dd10-41fc-bf25-1eb649faa71c None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:26.473751 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0b40a677-dd10-41fc-bf25-1eb649faa71c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:26.485481 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b40a677-dd10-41fc-bf25-1eb649faa71c None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:26.489173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea347e7e-9226-441d-82aa-9308157d7eaa service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:26.489800 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea347e7e-9226-441d-82aa-9308157d7eaa service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:26.490085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea347e7e-9226-441d-82aa-9308157d7eaa service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:26.490359 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea347e7e-9226-441d-82aa-9308157d7eaa service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:26.499388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea347e7e-9226-441d-82aa-9308157d7eaa service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:26.500735 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea347e7e-9226-441d-82aa-9308157d7eaa service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:26.510032 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1236/2484] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:26:26 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:26:31.862836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:INSERT=3,keystone:SELECT=65 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:36.505981 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=21 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:36.568137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef8de03-347b-4dec-be69-2307179bca92 service cinder] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:36.570872 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-4ef8de03-347b-4dec-be69-2307179bca92 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:36.703903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef8de03-347b-4dec-be69-2307179bca92 service cinder] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:36.716948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913686ed-5e6f-45ce-97c7-53745d9d091e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:36.717842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913686ed-5e6f-45ce-97c7-53745d9d091e service nova] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:36.718167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913686ed-5e6f-45ce-97c7-53745d9d091e service nova] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:36.718793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913686ed-5e6f-45ce-97c7-53745d9d091e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:36.721376 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-913686ed-5e6f-45ce-97c7-53745d9d091e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:36.739482 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913686ed-5e6f-45ce-97c7-53745d9d091e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:36.742181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913686ed-5e6f-45ce-97c7-53745d9d091e service nova] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:36.765977 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1249/2485] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:26:36 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 199 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:26:43.201985 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea347e7e-9226-441d-82aa-9308157d7eaa service cinder] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:43.203396 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ea347e7e-9226-441d-82aa-9308157d7eaa service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:43.258236 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea347e7e-9226-441d-82aa-9308157d7eaa service cinder] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:43.261769 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50bfa7e5-91eb-4704-b5d0-d60ad648f9dd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:43.262357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50bfa7e5-91eb-4704-b5d0-d60ad648f9dd service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:43.262633 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50bfa7e5-91eb-4704-b5d0-d60ad648f9dd service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:43.262900 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50bfa7e5-91eb-4704-b5d0-d60ad648f9dd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:43.264561 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-50bfa7e5-91eb-4704-b5d0-d60ad648f9dd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:43.326980 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50bfa7e5-91eb-4704-b5d0-d60ad648f9dd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:43.328434 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50bfa7e5-91eb-4704-b5d0-d60ad648f9dd service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:43.338833 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1237/2486] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:26:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:26:46.430443 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913686ed-5e6f-45ce-97c7-53745d9d091e service nova] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:46.431899 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-913686ed-5e6f-45ce-97c7-53745d9d091e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:46.446793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913686ed-5e6f-45ce-97c7-53745d9d091e service nova] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:46.451966 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8865c4a9-c8a3-460a-94e2-7b028c851024 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:46.452602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8865c4a9-c8a3-460a-94e2-7b028c851024 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:46.452880 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8865c4a9-c8a3-460a-94e2-7b028c851024 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:46.453148 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8865c4a9-c8a3-460a-94e2-7b028c851024 admin admin] PATH_INFO: `/v3/users/f3438d850ce541cb9027ea3d5fe7dbf3` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:46.454883 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8865c4a9-c8a3-460a-94e2-7b028c851024 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f3438d850ce541cb9027ea3d5fe7dbf3)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:46.456848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8865c4a9-c8a3-460a-94e2-7b028c851024 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:46.503386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8865c4a9-c8a3-460a-94e2-7b028c851024 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f3438d850ce541cb9027ea3d5fe7dbf3'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:46.510513 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8865c4a9-c8a3-460a-94e2-7b028c851024 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f3438d850ce541cb9027ea3d5fe7dbf3'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:46.511190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8865c4a9-c8a3-460a-94e2-7b028c851024 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f3438d850ce541cb9027ea3d5fe7dbf3'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:46.522888 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8865c4a9-c8a3-460a-94e2-7b028c851024 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f3438d850ce541cb9027ea3d5fe7dbf3'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:46.535926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8865c4a9-c8a3-460a-94e2-7b028c851024 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f3438d850ce541cb9027ea3d5fe7dbf3'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:46.542794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1250/2487] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:26:46 2022] DELETE /identity/v3/users/f3438d850ce541cb9027ea3d5fe7dbf3 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:46.734853 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50bfa7e5-91eb-4704-b5d0-d60ad648f9dd service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:46.749188 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50bfa7e5-91eb-4704-b5d0-d60ad648f9dd service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:46.815394 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807e216e-5718-429d-ac88-6d9dd97fa8d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:46.816039 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807e216e-5718-429d-ac88-6d9dd97fa8d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:46.816318 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807e216e-5718-429d-ac88-6d9dd97fa8d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:46.816589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807e216e-5718-429d-ac88-6d9dd97fa8d4 admin admin] PATH_INFO: `/v3/projects/1c6cc70cbd2b4c04864f33ebafa06b5a` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:46.817945 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807e216e-5718-429d-ac88-6d9dd97fa8d4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1c6cc70cbd2b4c04864f33ebafa06b5a)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:46.819836 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807e216e-5718-429d-ac88-6d9dd97fa8d4 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:46.869995 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-807e216e-5718-429d-ac88-6d9dd97fa8d4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1c6cc70cbd2b4c04864f33ebafa06b5a', 'request_id': 'req-807e216e-5718-429d-ac88-6d9dd97fa8d4'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:46.872075 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:46.877869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-807e216e-5718-429d-ac88-6d9dd97fa8d4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1c6cc70cbd2b4c04864f33ebafa06b5a', 'request_id': 'req-807e216e-5718-429d-ac88-6d9dd97fa8d4'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:46.884488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-807e216e-5718-429d-ac88-6d9dd97fa8d4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1c6cc70cbd2b4c04864f33ebafa06b5a', 'request_id': 'req-807e216e-5718-429d-ac88-6d9dd97fa8d4'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:46.896191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-807e216e-5718-429d-ac88-6d9dd97fa8d4 admin admin] The token cache is being invalidate because project 1c6cc70cbd2b4c04864f33ebafa06b5a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:26:46.896481 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-807e216e-5718-429d-ac88-6d9dd97fa8d4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:46.897227 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-807e216e-5718-429d-ac88-6d9dd97fa8d4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:46.898805 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1238/2488] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:26:46 2022] DELETE /identity/v3/projects/1c6cc70cbd2b4c04864f33ebafa06b5a => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:46.904129 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f82ed9e-b28f-42fd-ae50-c87281b7a83f None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:46.904409 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f82ed9e-b28f-42fd-ae50-c87281b7a83f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:46.904703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f82ed9e-b28f-42fd-ae50-c87281b7a83f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:46.927136 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-6f82ed9e-b28f-42fd-ae50-c87281b7a83f None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:46.928329 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6f82ed9e-b28f-42fd-ae50-c87281b7a83f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:46.981496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:46.994909 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1251/2489] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:47.003535 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edb0cbf-7340-4c69-b1f5-77f102537585 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:47.003826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edb0cbf-7340-4c69-b1f5-77f102537585 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:47.004214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edb0cbf-7340-4c69-b1f5-77f102537585 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:47.024238 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-9edb0cbf-7340-4c69-b1f5-77f102537585 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:47.025449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-9edb0cbf-7340-4c69-b1f5-77f102537585 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:47.033367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1239/2490] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:47.037828 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f82ed9e-b28f-42fd-ae50-c87281b7a83f None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:47.048237 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f82ed9e-b28f-42fd-ae50-c87281b7a83f None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:47.053134 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82864461-b647-4407-9715-e39ef151dd89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:47.053786 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82864461-b647-4407-9715-e39ef151dd89 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:47.054083 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82864461-b647-4407-9715-e39ef151dd89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:47.054339 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82864461-b647-4407-9715-e39ef151dd89 admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:47.054986 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82864461-b647-4407-9715-e39ef151dd89 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:26:47.055284 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82864461-b647-4407-9715-e39ef151dd89 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:47.056894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82864461-b647-4407-9715-e39ef151dd89 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:47.069815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1252/2491] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:26:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:26:47.074129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edb0cbf-7340-4c69-b1f5-77f102537585 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:47.084674 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edb0cbf-7340-4c69-b1f5-77f102537585 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:47.089229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e342f8-06d6-4348-9429-b87d142d947d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:47.089852 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e342f8-06d6-4348-9429-b87d142d947d admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:47.090131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e342f8-06d6-4348-9429-b87d142d947d admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:47.090403 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e342f8-06d6-4348-9429-b87d142d947d admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:47.091023 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93e342f8-06d6-4348-9429-b87d142d947d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:47.093006 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93e342f8-06d6-4348-9429-b87d142d947d admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:47.123353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1240/2492] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:26:47 2022] POST /identity/v3/projects => generated 405 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:26:47.127450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82864461-b647-4407-9715-e39ef151dd89 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:47.137295 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82864461-b647-4407-9715-e39ef151dd89 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:47.195076 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4a690e-d595-4299-9403-bc2adf1b2646 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:47.195738 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4a690e-d595-4299-9403-bc2adf1b2646 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:47.195955 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4a690e-d595-4299-9403-bc2adf1b2646 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:47.196223 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4a690e-d595-4299-9403-bc2adf1b2646 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:47.196854 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4a690e-d595-4299-9403-bc2adf1b2646 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:47.198771 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4a690e-d595-4299-9403-bc2adf1b2646 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:47.220739 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1253/2493] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:26:47 2022] POST /identity/v3/users => generated 303 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:26:47.224697 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e342f8-06d6-4348-9429-b87d142d947d admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:47.237044 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e342f8-06d6-4348-9429-b87d142d947d admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:47.241720 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c599cdb8-5505-4398-9fbf-7bab26df0a1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:47.242344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c599cdb8-5505-4398-9fbf-7bab26df0a1c admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:47.242624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c599cdb8-5505-4398-9fbf-7bab26df0a1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:47.242894 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c599cdb8-5505-4398-9fbf-7bab26df0a1c admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:47.243529 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c599cdb8-5505-4398-9fbf-7bab26df0a1c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:47.245100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c599cdb8-5505-4398-9fbf-7bab26df0a1c admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:47.255621 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1241/2494] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:47 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:47.259710 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4a690e-d595-4299-9403-bc2adf1b2646 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:47.269590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4a690e-d595-4299-9403-bc2adf1b2646 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:47.274206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c895a27-f80e-4556-adc4-d526ba596bdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:47.274793 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c895a27-f80e-4556-adc4-d526ba596bdc admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:47.275068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c895a27-f80e-4556-adc4-d526ba596bdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:47.275340 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c895a27-f80e-4556-adc4-d526ba596bdc admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:47.275971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c895a27-f80e-4556-adc4-d526ba596bdc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:47.277545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c895a27-f80e-4556-adc4-d526ba596bdc admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:47.289479 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1254/2495] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:47 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:47.293339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c599cdb8-5505-4398-9fbf-7bab26df0a1c admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:47.304139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c599cdb8-5505-4398-9fbf-7bab26df0a1c admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:47.308651 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6432b4dd-55c3-406a-aa0a-f8eb2cf8f3b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:47.309286 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6432b4dd-55c3-406a-aa0a-f8eb2cf8f3b9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:47.309563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6432b4dd-55c3-406a-aa0a-f8eb2cf8f3b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:47.309864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6432b4dd-55c3-406a-aa0a-f8eb2cf8f3b9 admin admin] PATH_INFO: `/v3/projects/1c32dbf98bb9458c8ee9484fde9ba309/users/67a2e872b76b4262be6e0a71de66737c/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:47.324176 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6432b4dd-55c3-406a-aa0a-f8eb2cf8f3b9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1c32dbf98bb9458c8ee9484fde9ba309, user_id=67a2e872b76b4262be6e0a71de66737c, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:47.326807 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6432b4dd-55c3-406a-aa0a-f8eb2cf8f3b9 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:47.339257 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1242/2496] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:26:47 2022] PUT /identity/v3/projects/1c32dbf98bb9458c8ee9484fde9ba309/users/67a2e872b76b4262be6e0a71de66737c/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:26:47.350065 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f78c76-070a-4d7d-9bb1-bd4ff1a23fbf None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:47.350363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f78c76-070a-4d7d-9bb1-bd4ff1a23fbf None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:47.350638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f78c76-070a-4d7d-9bb1-bd4ff1a23fbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:47.377854 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-10f78c76-070a-4d7d-9bb1-bd4ff1a23fbf None None] MFA Rules not processed for user `67a2e872b76b4262be6e0a71de66737c`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:47.378969 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-10f78c76-070a-4d7d-9bb1-bd4ff1a23fbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:47.441530 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1255/2497] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:47 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:47.446020 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6432b4dd-55c3-406a-aa0a-f8eb2cf8f3b9 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:47.507462 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6432b4dd-55c3-406a-aa0a-f8eb2cf8f3b9 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:47.512055 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd55e304-5665-4467-bf93-f48b6eada24d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:47.512644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd55e304-5665-4467-bf93-f48b6eada24d admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:47.512919 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd55e304-5665-4467-bf93-f48b6eada24d admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:47.513185 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd55e304-5665-4467-bf93-f48b6eada24d admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:47.513843 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd55e304-5665-4467-bf93-f48b6eada24d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:47.515740 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd55e304-5665-4467-bf93-f48b6eada24d admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:47.542164 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1243/2498] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:26:47 2022] POST /identity/v3/projects => generated 405 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:26:47.546451 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f78c76-070a-4d7d-9bb1-bd4ff1a23fbf None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:47.558007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f78c76-070a-4d7d-9bb1-bd4ff1a23fbf None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:47.604241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:47.613803 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08fa9e4c-910a-4bed-9fa4-4bfc26fa847b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:47.614400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fa9e4c-910a-4bed-9fa4-4bfc26fa847b admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:47.614672 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fa9e4c-910a-4bed-9fa4-4bfc26fa847b admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:47.614938 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fa9e4c-910a-4bed-9fa4-4bfc26fa847b admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:47.615582 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08fa9e4c-910a-4bed-9fa4-4bfc26fa847b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:47.617462 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08fa9e4c-910a-4bed-9fa4-4bfc26fa847b admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:47.638025 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1256/2499] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:26:47 2022] POST /identity/v3/users => generated 309 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:26:47.642541 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd55e304-5665-4467-bf93-f48b6eada24d admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:47.652806 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd55e304-5665-4467-bf93-f48b6eada24d admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:47.658223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5538ac88-f55d-4c5e-b3dd-2b0ea447f080 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:47.658814 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5538ac88-f55d-4c5e-b3dd-2b0ea447f080 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:47.659090 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5538ac88-f55d-4c5e-b3dd-2b0ea447f080 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:47.659358 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5538ac88-f55d-4c5e-b3dd-2b0ea447f080 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:47.659991 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5538ac88-f55d-4c5e-b3dd-2b0ea447f080 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:47.661586 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5538ac88-f55d-4c5e-b3dd-2b0ea447f080 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:47.672132 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1244/2500] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:47 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:47.676117 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08fa9e4c-910a-4bed-9fa4-4bfc26fa847b admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:47.686002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08fa9e4c-910a-4bed-9fa4-4bfc26fa847b admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:47.690621 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dff3204-b193-41a0-84d3-1041a68fd69d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:47.691255 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dff3204-b193-41a0-84d3-1041a68fd69d admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:47.691549 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dff3204-b193-41a0-84d3-1041a68fd69d admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:47.691818 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dff3204-b193-41a0-84d3-1041a68fd69d admin admin] PATH_INFO: `/v3/projects/85f605240d5845b6a9aa1583e6314e48/users/95f4138cf33f4b918169fcc37ec7471e/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:47.706556 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dff3204-b193-41a0-84d3-1041a68fd69d admin admin] RBAC: Authorizing `identity:create_grant(project_id=85f605240d5845b6a9aa1583e6314e48, user_id=95f4138cf33f4b918169fcc37ec7471e, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:47.709161 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dff3204-b193-41a0-84d3-1041a68fd69d admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:47.725819 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1257/2501] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:26:47 2022] PUT /identity/v3/projects/85f605240d5845b6a9aa1583e6314e48/users/95f4138cf33f4b918169fcc37ec7471e/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:47.727495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5538ac88-f55d-4c5e-b3dd-2b0ea447f080 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:47.740850 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5538ac88-f55d-4c5e-b3dd-2b0ea447f080 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:47.790063 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:47.801206 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50868fe-7e22-44e8-be92-0bbdfad1321e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:47.802451 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50868fe-7e22-44e8-be92-0bbdfad1321e admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:47.802743 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50868fe-7e22-44e8-be92-0bbdfad1321e admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:47.803044 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50868fe-7e22-44e8-be92-0bbdfad1321e admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:47.803841 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a50868fe-7e22-44e8-be92-0bbdfad1321e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:47.805810 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a50868fe-7e22-44e8-be92-0bbdfad1321e admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:47.816379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1245/2502] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:47 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:47.820635 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dff3204-b193-41a0-84d3-1041a68fd69d admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:47.831136 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dff3204-b193-41a0-84d3-1041a68fd69d admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:47.835633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0850940-6a29-4fa3-85d3-aaf924f4d611 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:47.836271 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0850940-6a29-4fa3-85d3-aaf924f4d611 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:47.836550 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0850940-6a29-4fa3-85d3-aaf924f4d611 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:47.836879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0850940-6a29-4fa3-85d3-aaf924f4d611 admin admin] PATH_INFO: `/v3/projects/85f605240d5845b6a9aa1583e6314e48/users/95f4138cf33f4b918169fcc37ec7471e/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:47.838904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0850940-6a29-4fa3-85d3-aaf924f4d611 admin admin] RBAC: Authorizing `identity:create_grant(project_id=85f605240d5845b6a9aa1583e6314e48, user_id=95f4138cf33f4b918169fcc37ec7471e, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:47.841377 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0850940-6a29-4fa3-85d3-aaf924f4d611 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:47.852040 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1258/2503] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:26:47 2022] PUT /identity/v3/projects/85f605240d5845b6a9aa1583e6314e48/users/95f4138cf33f4b918169fcc37ec7471e/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:47.862643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4656a40a-baf5-4321-9944-b6a9294310c4 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:47.863051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4656a40a-baf5-4321-9944-b6a9294310c4 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:47.863323 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4656a40a-baf5-4321-9944-b6a9294310c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:47.891113 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-4656a40a-baf5-4321-9944-b6a9294310c4 None None] MFA Rules not processed for user `95f4138cf33f4b918169fcc37ec7471e`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:47.892845 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-4656a40a-baf5-4321-9944-b6a9294310c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:47.965944 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1246/2504] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:47 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:47.977441 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0850940-6a29-4fa3-85d3-aaf924f4d611 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:47.979319 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c0850940-6a29-4fa3-85d3-aaf924f4d611 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:48.040256 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0850940-6a29-4fa3-85d3-aaf924f4d611 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:48.044833 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215128f8-2416-4548-a064-ec97f35d2fc7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:48.048279 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215128f8-2416-4548-a064-ec97f35d2fc7 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:48.048279 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215128f8-2416-4548-a064-ec97f35d2fc7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:48.048279 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215128f8-2416-4548-a064-ec97f35d2fc7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:48.055872 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215128f8-2416-4548-a064-ec97f35d2fc7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:48.057491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215128f8-2416-4548-a064-ec97f35d2fc7 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:48.067820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1259/2505] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:26:47 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:26:48.210999 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4656a40a-baf5-4321-9944-b6a9294310c4 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:48.278649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4656a40a-baf5-4321-9944-b6a9294310c4 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:48.283196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c302db-8865-418d-a2d0-70349fc6efb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:48.283819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c302db-8865-418d-a2d0-70349fc6efb9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:48.284098 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c302db-8865-418d-a2d0-70349fc6efb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:48.284364 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c302db-8865-418d-a2d0-70349fc6efb9 admin admin] PATH_INFO: `/v3/users/67a2e872b76b4262be6e0a71de66737c` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:48.285491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c302db-8865-418d-a2d0-70349fc6efb9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=67a2e872b76b4262be6e0a71de66737c)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:48.287526 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c302db-8865-418d-a2d0-70349fc6efb9 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:48.336983 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-63c302db-8865-418d-a2d0-70349fc6efb9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '67a2e872b76b4262be6e0a71de66737c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:48.343931 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-63c302db-8865-418d-a2d0-70349fc6efb9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '67a2e872b76b4262be6e0a71de66737c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:48.344613 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-63c302db-8865-418d-a2d0-70349fc6efb9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '67a2e872b76b4262be6e0a71de66737c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:48.351282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-63c302db-8865-418d-a2d0-70349fc6efb9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '67a2e872b76b4262be6e0a71de66737c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:48.359044 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:48.364606 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-63c302db-8865-418d-a2d0-70349fc6efb9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '67a2e872b76b4262be6e0a71de66737c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:48.371454 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1247/2506] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:26:48 2022] DELETE /identity/v3/users/67a2e872b76b4262be6e0a71de66737c => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:48.376482 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3046d0-ecf0-485b-8a23-4ac57f2ee82a None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:48.376792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3046d0-ecf0-485b-8a23-4ac57f2ee82a None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:48.377061 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3046d0-ecf0-485b-8a23-4ac57f2ee82a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:48.395792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-ca3046d0-ecf0-485b-8a23-4ac57f2ee82a None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:48.396927 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ca3046d0-ecf0-485b-8a23-4ac57f2ee82a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:48.455587 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1260/2507] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:48.466164 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c302db-8865-418d-a2d0-70349fc6efb9 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:48.467548 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-63c302db-8865-418d-a2d0-70349fc6efb9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:48.525327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c302db-8865-418d-a2d0-70349fc6efb9 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:48.530139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc069fb-e160-4c71-9d2e-b7db7fd1bdd5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:48.531098 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc069fb-e160-4c71-9d2e-b7db7fd1bdd5 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:48.531564 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc069fb-e160-4c71-9d2e-b7db7fd1bdd5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:48.532050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc069fb-e160-4c71-9d2e-b7db7fd1bdd5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:48.541952 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc069fb-e160-4c71-9d2e-b7db7fd1bdd5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:48.543801 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc069fb-e160-4c71-9d2e-b7db7fd1bdd5 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:48.554650 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1248/2508] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:26:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:26:48.817562 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3046d0-ecf0-485b-8a23-4ac57f2ee82a None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:48.819021 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ca3046d0-ecf0-485b-8a23-4ac57f2ee82a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:48.831149 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3046d0-ecf0-485b-8a23-4ac57f2ee82a None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:48.835721 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bce84c2-5e3c-4bea-9eb8-f3e2544227b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:48.836350 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bce84c2-5e3c-4bea-9eb8-f3e2544227b6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:48.836625 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bce84c2-5e3c-4bea-9eb8-f3e2544227b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:48.836895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bce84c2-5e3c-4bea-9eb8-f3e2544227b6 admin admin] PATH_INFO: `/v3/projects/1c32dbf98bb9458c8ee9484fde9ba309` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:48.838158 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bce84c2-5e3c-4bea-9eb8-f3e2544227b6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1c32dbf98bb9458c8ee9484fde9ba309)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:48.840030 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bce84c2-5e3c-4bea-9eb8-f3e2544227b6 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:48.882623 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=92,keystone:INSERT=5,keystone:DELETE=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:48.886786 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-4bce84c2-5e3c-4bea-9eb8-f3e2544227b6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1c32dbf98bb9458c8ee9484fde9ba309', 'request_id': 'req-4bce84c2-5e3c-4bea-9eb8-f3e2544227b6'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:48.893724 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-4bce84c2-5e3c-4bea-9eb8-f3e2544227b6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1c32dbf98bb9458c8ee9484fde9ba309', 'request_id': 'req-4bce84c2-5e3c-4bea-9eb8-f3e2544227b6'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:48.899878 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-4bce84c2-5e3c-4bea-9eb8-f3e2544227b6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1c32dbf98bb9458c8ee9484fde9ba309', 'request_id': 'req-4bce84c2-5e3c-4bea-9eb8-f3e2544227b6'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:48.912676 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-4bce84c2-5e3c-4bea-9eb8-f3e2544227b6 admin admin] The token cache is being invalidate because project 1c32dbf98bb9458c8ee9484fde9ba309 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:26:48.912958 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-4bce84c2-5e3c-4bea-9eb8-f3e2544227b6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:48.913588 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-4bce84c2-5e3c-4bea-9eb8-f3e2544227b6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:48.915369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1261/2509] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:26:48 2022] DELETE /identity/v3/projects/1c32dbf98bb9458c8ee9484fde9ba309 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:48.919305 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc069fb-e160-4c71-9d2e-b7db7fd1bdd5 service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:48.934736 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc069fb-e160-4c71-9d2e-b7db7fd1bdd5 service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:48.991652 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7699c7-d6a8-4a3e-9230-01e538efb047 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:48.992251 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7699c7-d6a8-4a3e-9230-01e538efb047 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:48.992527 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7699c7-d6a8-4a3e-9230-01e538efb047 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:48.992799 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7699c7-d6a8-4a3e-9230-01e538efb047 admin admin] PATH_INFO: `/v3/users/95f4138cf33f4b918169fcc37ec7471e` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:48.993915 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7699c7-d6a8-4a3e-9230-01e538efb047 admin admin] RBAC: Authorizing `identity:delete_user(user_id=95f4138cf33f4b918169fcc37ec7471e)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:48.995861 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7699c7-d6a8-4a3e-9230-01e538efb047 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:49.040769 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-1a7699c7-d6a8-4a3e-9230-01e538efb047 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '95f4138cf33f4b918169fcc37ec7471e'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:49.047867 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-1a7699c7-d6a8-4a3e-9230-01e538efb047 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '95f4138cf33f4b918169fcc37ec7471e'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:49.048631 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-1a7699c7-d6a8-4a3e-9230-01e538efb047 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '95f4138cf33f4b918169fcc37ec7471e'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:49.056181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-1a7699c7-d6a8-4a3e-9230-01e538efb047 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '95f4138cf33f4b918169fcc37ec7471e'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:49.068547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-1a7699c7-d6a8-4a3e-9230-01e538efb047 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '95f4138cf33f4b918169fcc37ec7471e'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:49.074289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1249/2510] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:26:48 2022] DELETE /identity/v3/users/95f4138cf33f4b918169fcc37ec7471e => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:49.349873 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bce84c2-5e3c-4bea-9eb8-f3e2544227b6 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:49.354459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-4bce84c2-5e3c-4bea-9eb8-f3e2544227b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:49.378480 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bce84c2-5e3c-4bea-9eb8-f3e2544227b6 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:49.451460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32bee83-b080-40cc-a6ed-845af7bec540 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:49.452096 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32bee83-b080-40cc-a6ed-845af7bec540 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:49.452372 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32bee83-b080-40cc-a6ed-845af7bec540 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:49.452641 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32bee83-b080-40cc-a6ed-845af7bec540 admin admin] PATH_INFO: `/v3/projects/85f605240d5845b6a9aa1583e6314e48` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:49.453814 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32bee83-b080-40cc-a6ed-845af7bec540 admin admin] RBAC: Authorizing `identity:delete_project(project_id=85f605240d5845b6a9aa1583e6314e48)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:49.456390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32bee83-b080-40cc-a6ed-845af7bec540 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:49.501977 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-c32bee83-b080-40cc-a6ed-845af7bec540 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '85f605240d5845b6a9aa1583e6314e48', 'request_id': 'req-c32bee83-b080-40cc-a6ed-845af7bec540'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:49.507340 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-c32bee83-b080-40cc-a6ed-845af7bec540 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '85f605240d5845b6a9aa1583e6314e48', 'request_id': 'req-c32bee83-b080-40cc-a6ed-845af7bec540'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:49.513281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-c32bee83-b080-40cc-a6ed-845af7bec540 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '85f605240d5845b6a9aa1583e6314e48', 'request_id': 'req-c32bee83-b080-40cc-a6ed-845af7bec540'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:49.526789 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-c32bee83-b080-40cc-a6ed-845af7bec540 admin admin] The token cache is being invalidate because project 85f605240d5845b6a9aa1583e6314e48 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:26:49.527194 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-c32bee83-b080-40cc-a6ed-845af7bec540 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:49.527879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-c32bee83-b080-40cc-a6ed-845af7bec540 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:49.529532 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1262/2511] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:26:49 2022] DELETE /identity/v3/projects/85f605240d5845b6a9aa1583e6314e48 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:49.534639 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e1ca1b-e195-4ad7-9895-007fed22acf7 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:49.534928 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e1ca1b-e195-4ad7-9895-007fed22acf7 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:49.535205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e1ca1b-e195-4ad7-9895-007fed22acf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:49.562040 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-81e1ca1b-e195-4ad7-9895-007fed22acf7 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:49.564040 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-81e1ca1b-e195-4ad7-9895-007fed22acf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:49.573088 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:49.628865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1250/2512] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:26:49.638498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6f16b0-c190-4463-89a7-24b096232a8e None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:49.639285 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6f16b0-c190-4463-89a7-24b096232a8e None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:49.639285 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6f16b0-c190-4463-89a7-24b096232a8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:49.660020 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-ad6f16b0-c190-4463-89a7-24b096232a8e None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:49.661279 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ad6f16b0-c190-4463-89a7-24b096232a8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:49.669397 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1263/2513] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:26:49.674020 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e1ca1b-e195-4ad7-9895-007fed22acf7 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:49.683646 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e1ca1b-e195-4ad7-9895-007fed22acf7 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:49.688089 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac30c930-eed4-401b-a324-ec4ef2387c87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:49.689326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac30c930-eed4-401b-a324-ec4ef2387c87 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:49.689608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac30c930-eed4-401b-a324-ec4ef2387c87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:49.689901 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac30c930-eed4-401b-a324-ec4ef2387c87 admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:49.690546 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac30c930-eed4-401b-a324-ec4ef2387c87 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:26:49.690849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac30c930-eed4-401b-a324-ec4ef2387c87 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:49.692420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac30c930-eed4-401b-a324-ec4ef2387c87 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:49.703777 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1251/2514] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:26:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:26:49.707950 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6f16b0-c190-4463-89a7-24b096232a8e None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:49.717311 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6f16b0-c190-4463-89a7-24b096232a8e None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:49.722180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1e9934-6530-4470-b6fc-4c005623d030 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:49.722776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1e9934-6530-4470-b6fc-4c005623d030 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:49.723056 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1e9934-6530-4470-b6fc-4c005623d030 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:49.723328 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1e9934-6530-4470-b6fc-4c005623d030 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:49.723988 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1e9934-6530-4470-b6fc-4c005623d030 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:49.725913 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1e9934-6530-4470-b6fc-4c005623d030 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:49.755184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1264/2515] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:26:49 2022] POST /identity/v3/projects => generated 373 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:26:49.759488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac30c930-eed4-401b-a324-ec4ef2387c87 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:49.769386 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac30c930-eed4-401b-a324-ec4ef2387c87 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:49.825538 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8eadd24-f316-4e02-8cd4-ad8d0750842b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:49.826162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eadd24-f316-4e02-8cd4-ad8d0750842b admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:49.826443 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eadd24-f316-4e02-8cd4-ad8d0750842b admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:49.826717 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eadd24-f316-4e02-8cd4-ad8d0750842b admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:49.827333 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8eadd24-f316-4e02-8cd4-ad8d0750842b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:49.829316 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8eadd24-f316-4e02-8cd4-ad8d0750842b admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:49.849368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1252/2516] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:26:49 2022] POST /identity/v3/users => generated 287 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:26:49.853482 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1e9934-6530-4470-b6fc-4c005623d030 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:49.864141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1e9934-6530-4470-b6fc-4c005623d030 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:49.868922 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5e4c7c-a835-48d1-b2fc-0e004e8e5f74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:49.869544 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5e4c7c-a835-48d1-b2fc-0e004e8e5f74 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:49.869848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5e4c7c-a835-48d1-b2fc-0e004e8e5f74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:49.870121 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5e4c7c-a835-48d1-b2fc-0e004e8e5f74 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:49.870758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5e4c7c-a835-48d1-b2fc-0e004e8e5f74 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:49.872343 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5e4c7c-a835-48d1-b2fc-0e004e8e5f74 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:49.882841 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1265/2517] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:49 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:49.886546 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8eadd24-f316-4e02-8cd4-ad8d0750842b admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:49.896524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8eadd24-f316-4e02-8cd4-ad8d0750842b admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:49.901145 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff772da5-ff6b-4413-b77a-2ff7e79e9a7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:49.901733 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff772da5-ff6b-4413-b77a-2ff7e79e9a7e admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:49.902041 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff772da5-ff6b-4413-b77a-2ff7e79e9a7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:49.902309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff772da5-ff6b-4413-b77a-2ff7e79e9a7e admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:49.902944 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff772da5-ff6b-4413-b77a-2ff7e79e9a7e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:49.904502 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff772da5-ff6b-4413-b77a-2ff7e79e9a7e admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:49.915824 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1253/2518] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:49 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:26:49.920072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5e4c7c-a835-48d1-b2fc-0e004e8e5f74 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:49.930787 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5e4c7c-a835-48d1-b2fc-0e004e8e5f74 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:49.935411 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca7d7e5-02d3-4aae-8447-01b712d88c39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:49.936153 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca7d7e5-02d3-4aae-8447-01b712d88c39 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:49.936429 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca7d7e5-02d3-4aae-8447-01b712d88c39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:49.936705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca7d7e5-02d3-4aae-8447-01b712d88c39 admin admin] PATH_INFO: `/v3/projects/ae2f2bea79ab48c5bd28339fdac86fb3/users/0c510124846b47598c04441bf5dd151d/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:49.952498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca7d7e5-02d3-4aae-8447-01b712d88c39 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ae2f2bea79ab48c5bd28339fdac86fb3, user_id=0c510124846b47598c04441bf5dd151d, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:49.955585 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca7d7e5-02d3-4aae-8447-01b712d88c39 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:49.963348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=85,keystone:DELETE=11,keystone:INSERT=4 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:49.969148 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1266/2519] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:26:49 2022] PUT /identity/v3/projects/ae2f2bea79ab48c5bd28339fdac86fb3/users/0c510124846b47598c04441bf5dd151d/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:26:49.979352 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322c0282-36f9-477a-8240-71c5700346f1 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:49.979820 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322c0282-36f9-477a-8240-71c5700346f1 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:49.979913 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322c0282-36f9-477a-8240-71c5700346f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:50.008092 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-322c0282-36f9-477a-8240-71c5700346f1 None None] MFA Rules not processed for user `0c510124846b47598c04441bf5dd151d`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:50.009304 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-322c0282-36f9-477a-8240-71c5700346f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:50.071760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1254/2520] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:49 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:50.076197 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca7d7e5-02d3-4aae-8447-01b712d88c39 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:50.135708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca7d7e5-02d3-4aae-8447-01b712d88c39 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:50.140261 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824f026d-069a-4549-8103-9f1eeaa99c80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:50.140851 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824f026d-069a-4549-8103-9f1eeaa99c80 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:50.141127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824f026d-069a-4549-8103-9f1eeaa99c80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:50.141449 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824f026d-069a-4549-8103-9f1eeaa99c80 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:50.142101 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824f026d-069a-4549-8103-9f1eeaa99c80 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:50.143974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824f026d-069a-4549-8103-9f1eeaa99c80 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:50.171157 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1267/2521] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:26:50 2022] POST /identity/v3/projects => generated 373 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:26:50.175847 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322c0282-36f9-477a-8240-71c5700346f1 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:50.187315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322c0282-36f9-477a-8240-71c5700346f1 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:50.191907 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:50.245802 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560e5226-e9a0-4c42-8784-db7ec1dc8f9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:50.246407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560e5226-e9a0-4c42-8784-db7ec1dc8f9e admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:50.246686 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560e5226-e9a0-4c42-8784-db7ec1dc8f9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:50.246956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560e5226-e9a0-4c42-8784-db7ec1dc8f9e admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:50.247828 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560e5226-e9a0-4c42-8784-db7ec1dc8f9e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:50.249726 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560e5226-e9a0-4c42-8784-db7ec1dc8f9e admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:50.272011 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1255/2522] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:26:50 2022] POST /identity/v3/users => generated 293 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:26:50.275969 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824f026d-069a-4549-8103-9f1eeaa99c80 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:50.288081 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824f026d-069a-4549-8103-9f1eeaa99c80 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:50.293770 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33eba0ba-dc2d-490c-8280-2cd3c991b095 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:50.294379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33eba0ba-dc2d-490c-8280-2cd3c991b095 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:50.294677 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33eba0ba-dc2d-490c-8280-2cd3c991b095 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:50.294944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33eba0ba-dc2d-490c-8280-2cd3c991b095 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:50.295603 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33eba0ba-dc2d-490c-8280-2cd3c991b095 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:50.297216 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33eba0ba-dc2d-490c-8280-2cd3c991b095 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:50.308615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1268/2523] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:50 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:50.312731 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560e5226-e9a0-4c42-8784-db7ec1dc8f9e admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:50.322856 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560e5226-e9a0-4c42-8784-db7ec1dc8f9e admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:50.327432 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a427b2-b3ba-4287-bc99-ef9af6027de7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:50.328066 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a427b2-b3ba-4287-bc99-ef9af6027de7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:50.328346 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a427b2-b3ba-4287-bc99-ef9af6027de7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:50.328618 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a427b2-b3ba-4287-bc99-ef9af6027de7 admin admin] PATH_INFO: `/v3/projects/7614a1b21d144efeb90a6e7ced0b9ed0/users/ffd308127d1946debcb689908632c7b0/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:50.343096 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a427b2-b3ba-4287-bc99-ef9af6027de7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7614a1b21d144efeb90a6e7ced0b9ed0, user_id=ffd308127d1946debcb689908632c7b0, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:50.346267 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a427b2-b3ba-4287-bc99-ef9af6027de7 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:50.358331 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1256/2524] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:26:50 2022] PUT /identity/v3/projects/7614a1b21d144efeb90a6e7ced0b9ed0/users/ffd308127d1946debcb689908632c7b0/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:50.362910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33eba0ba-dc2d-490c-8280-2cd3c991b095 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:50.372998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33eba0ba-dc2d-490c-8280-2cd3c991b095 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:50.429680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e07e590-f204-4216-9c5a-82b72b5f538a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:50.430243 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e07e590-f204-4216-9c5a-82b72b5f538a admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:50.430519 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e07e590-f204-4216-9c5a-82b72b5f538a admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:50.430789 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e07e590-f204-4216-9c5a-82b72b5f538a admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:50.431425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e07e590-f204-4216-9c5a-82b72b5f538a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:50.432993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e07e590-f204-4216-9c5a-82b72b5f538a admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:50.442494 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1269/2525] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:50 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:50.446584 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a427b2-b3ba-4287-bc99-ef9af6027de7 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:50.456993 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a427b2-b3ba-4287-bc99-ef9af6027de7 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:50.461512 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc55875-dc0b-44b9-b59c-f5fb7a3499d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:50.462320 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc55875-dc0b-44b9-b59c-f5fb7a3499d4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:50.462600 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc55875-dc0b-44b9-b59c-f5fb7a3499d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:50.462891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc55875-dc0b-44b9-b59c-f5fb7a3499d4 admin admin] PATH_INFO: `/v3/projects/7614a1b21d144efeb90a6e7ced0b9ed0/users/ffd308127d1946debcb689908632c7b0/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:50.464770 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc55875-dc0b-44b9-b59c-f5fb7a3499d4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7614a1b21d144efeb90a6e7ced0b9ed0, user_id=ffd308127d1946debcb689908632c7b0, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:50.467384 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc55875-dc0b-44b9-b59c-f5fb7a3499d4 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:50.478424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1257/2526] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:26:50 2022] PUT /identity/v3/projects/7614a1b21d144efeb90a6e7ced0b9ed0/users/ffd308127d1946debcb689908632c7b0/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:50.488349 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40f3e23-375e-40d4-bf79-66f8e84008ca None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:50.488865 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40f3e23-375e-40d4-bf79-66f8e84008ca None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:50.488865 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40f3e23-375e-40d4-bf79-66f8e84008ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:50.494798 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc55875-dc0b-44b9-b59c-f5fb7a3499d4 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:50.496436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-fbc55875-dc0b-44b9-b59c-f5fb7a3499d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:50.518045 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-a40f3e23-375e-40d4-bf79-66f8e84008ca None None] MFA Rules not processed for user `ffd308127d1946debcb689908632c7b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:50.519314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a40f3e23-375e-40d4-bf79-66f8e84008ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:50.562699 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc55875-dc0b-44b9-b59c-f5fb7a3499d4 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:50.567290 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d857ccfd-5e85-41ac-82f9-1f8f636ad03e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:50.567905 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d857ccfd-5e85-41ac-82f9-1f8f636ad03e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:50.568182 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d857ccfd-5e85-41ac-82f9-1f8f636ad03e admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:50.568450 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d857ccfd-5e85-41ac-82f9-1f8f636ad03e admin admin] PATH_INFO: `/v3/users/36c8cb515abf41b291ea187c8d0660cf` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:50.569555 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d857ccfd-5e85-41ac-82f9-1f8f636ad03e admin admin] RBAC: Authorizing `identity:delete_user(user_id=36c8cb515abf41b291ea187c8d0660cf)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:50.571478 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d857ccfd-5e85-41ac-82f9-1f8f636ad03e admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:50.581840 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:50.597609 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1270/2527] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:50 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:50.606913 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40f3e23-375e-40d4-bf79-66f8e84008ca None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:50.608374 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a40f3e23-375e-40d4-bf79-66f8e84008ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:50.630346 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-d857ccfd-5e85-41ac-82f9-1f8f636ad03e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '36c8cb515abf41b291ea187c8d0660cf'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:50.637289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-d857ccfd-5e85-41ac-82f9-1f8f636ad03e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '36c8cb515abf41b291ea187c8d0660cf'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:50.638031 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-d857ccfd-5e85-41ac-82f9-1f8f636ad03e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '36c8cb515abf41b291ea187c8d0660cf'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:50.646692 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-d857ccfd-5e85-41ac-82f9-1f8f636ad03e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '36c8cb515abf41b291ea187c8d0660cf'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:50.647944 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=86,keystone:INSERT=6,keystone:DELETE=8 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:50.662428 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-d857ccfd-5e85-41ac-82f9-1f8f636ad03e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '36c8cb515abf41b291ea187c8d0660cf'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:50.668625 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1258/2528] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:26:50 2022] DELETE /identity/v3/users/36c8cb515abf41b291ea187c8d0660cf => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:26:50.670628 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40f3e23-375e-40d4-bf79-66f8e84008ca None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:50.673118 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1496c6-90f1-4f07-b5af-d5b3216b5ae9 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:50.673402 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1496c6-90f1-4f07-b5af-d5b3216b5ae9 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:50.673675 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1496c6-90f1-4f07-b5af-d5b3216b5ae9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:50.692010 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-2d1496c6-90f1-4f07-b5af-d5b3216b5ae9 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:50.693135 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-2d1496c6-90f1-4f07-b5af-d5b3216b5ae9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:50.718365 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8475d23b-f038-46ed-9449-65318a6283e0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:50.718976 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8475d23b-f038-46ed-9449-65318a6283e0 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:50.719250 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8475d23b-f038-46ed-9449-65318a6283e0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:50.720146 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8475d23b-f038-46ed-9449-65318a6283e0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:50.721938 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-8475d23b-f038-46ed-9449-65318a6283e0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:50.752658 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1259/2529] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:26:50.762724 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1496c6-90f1-4f07-b5af-d5b3216b5ae9 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:50.764065 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-2d1496c6-90f1-4f07-b5af-d5b3216b5ae9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:50.793270 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8475d23b-f038-46ed-9449-65318a6283e0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:50.794733 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8475d23b-f038-46ed-9449-65318a6283e0 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:50.806314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1271/2530] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:26:50 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:26:50.823587 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1496c6-90f1-4f07-b5af-d5b3216b5ae9 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:50.827465 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e01d0ab-65ef-4422-8437-6364edc55b66 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:50.828081 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e01d0ab-65ef-4422-8437-6364edc55b66 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:50.828364 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e01d0ab-65ef-4422-8437-6364edc55b66 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:50.828639 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e01d0ab-65ef-4422-8437-6364edc55b66 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:50.838576 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e01d0ab-65ef-4422-8437-6364edc55b66 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:50.839994 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e01d0ab-65ef-4422-8437-6364edc55b66 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:50.849601 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1260/2531] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:26:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:26:50.881625 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8475d23b-f038-46ed-9449-65318a6283e0 service cinder] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:50.883125 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-8475d23b-f038-46ed-9449-65318a6283e0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:50.894527 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8475d23b-f038-46ed-9449-65318a6283e0 service cinder] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:50.898232 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64bb067-6236-4dd1-874c-08b04073b82e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:50.898837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64bb067-6236-4dd1-874c-08b04073b82e service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:50.899149 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64bb067-6236-4dd1-874c-08b04073b82e service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:50.899375 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64bb067-6236-4dd1-874c-08b04073b82e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:50.901272 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c64bb067-6236-4dd1-874c-08b04073b82e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:50.965380 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c64bb067-6236-4dd1-874c-08b04073b82e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:50.967054 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c64bb067-6236-4dd1-874c-08b04073b82e service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:50.970689 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:50.977566 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1272/2532] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:26:50 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:26:51.129447 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e01d0ab-65ef-4422-8437-6364edc55b66 service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:51.130966 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-6e01d0ab-65ef-4422-8437-6364edc55b66 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:51.146530 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e01d0ab-65ef-4422-8437-6364edc55b66 service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:51.152559 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af540a22-9810-4a72-af8a-bd703c7ed67b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:51.153196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af540a22-9810-4a72-af8a-bd703c7ed67b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:51.153477 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af540a22-9810-4a72-af8a-bd703c7ed67b admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:51.153819 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af540a22-9810-4a72-af8a-bd703c7ed67b admin admin] PATH_INFO: `/v3/projects/80ffbd2c0065498eabe1a93e2e3e0111` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:51.154966 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af540a22-9810-4a72-af8a-bd703c7ed67b admin admin] RBAC: Authorizing `identity:delete_project(project_id=80ffbd2c0065498eabe1a93e2e3e0111)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:51.157056 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af540a22-9810-4a72-af8a-bd703c7ed67b admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:51.207012 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-af540a22-9810-4a72-af8a-bd703c7ed67b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '80ffbd2c0065498eabe1a93e2e3e0111', 'request_id': 'req-af540a22-9810-4a72-af8a-bd703c7ed67b'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:51.212453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-af540a22-9810-4a72-af8a-bd703c7ed67b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '80ffbd2c0065498eabe1a93e2e3e0111', 'request_id': 'req-af540a22-9810-4a72-af8a-bd703c7ed67b'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:51.218538 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-af540a22-9810-4a72-af8a-bd703c7ed67b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '80ffbd2c0065498eabe1a93e2e3e0111', 'request_id': 'req-af540a22-9810-4a72-af8a-bd703c7ed67b'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:51.231385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-af540a22-9810-4a72-af8a-bd703c7ed67b admin admin] The token cache is being invalidate because project 80ffbd2c0065498eabe1a93e2e3e0111 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:26:51.231709 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-af540a22-9810-4a72-af8a-bd703c7ed67b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:51.232332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-af540a22-9810-4a72-af8a-bd703c7ed67b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:51.233871 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1261/2533] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:26:51 2022] DELETE /identity/v3/projects/80ffbd2c0065498eabe1a93e2e3e0111 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:51.238527 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64bb067-6236-4dd1-874c-08b04073b82e service cinder] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:51.248975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64bb067-6236-4dd1-874c-08b04073b82e service cinder] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:51.304184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190d344d-5da8-4fe9-8714-a77a8c3a7393 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:51.304798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190d344d-5da8-4fe9-8714-a77a8c3a7393 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:51.305072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190d344d-5da8-4fe9-8714-a77a8c3a7393 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:51.305346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190d344d-5da8-4fe9-8714-a77a8c3a7393 admin admin] PATH_INFO: `/v3/users/b9feb2405c7b4194a87b075f16fcac6c` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:51.306479 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190d344d-5da8-4fe9-8714-a77a8c3a7393 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b9feb2405c7b4194a87b075f16fcac6c)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:51.308421 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190d344d-5da8-4fe9-8714-a77a8c3a7393 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:51.352581 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-190d344d-5da8-4fe9-8714-a77a8c3a7393 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b9feb2405c7b4194a87b075f16fcac6c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:51.359403 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-190d344d-5da8-4fe9-8714-a77a8c3a7393 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b9feb2405c7b4194a87b075f16fcac6c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:51.360601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-190d344d-5da8-4fe9-8714-a77a8c3a7393 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b9feb2405c7b4194a87b075f16fcac6c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:51.368181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-190d344d-5da8-4fe9-8714-a77a8c3a7393 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b9feb2405c7b4194a87b075f16fcac6c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:51.380730 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-190d344d-5da8-4fe9-8714-a77a8c3a7393 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b9feb2405c7b4194a87b075f16fcac6c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:51.386583 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1273/2534] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:26:51 2022] DELETE /identity/v3/users/b9feb2405c7b4194a87b075f16fcac6c => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:51.652355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af540a22-9810-4a72-af8a-bd703c7ed67b admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:51.653820 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-af540a22-9810-4a72-af8a-bd703c7ed67b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:51.667355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af540a22-9810-4a72-af8a-bd703c7ed67b admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:51.671144 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:51.727818 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54b9657-5838-48cb-b5cd-db1ea1bec551 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:51.728461 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54b9657-5838-48cb-b5cd-db1ea1bec551 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:51.728736 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54b9657-5838-48cb-b5cd-db1ea1bec551 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:51.729559 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54b9657-5838-48cb-b5cd-db1ea1bec551 admin admin] PATH_INFO: `/v3/projects/6ebc46fe189d440599fb32a60fa430c2` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:51.730155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54b9657-5838-48cb-b5cd-db1ea1bec551 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6ebc46fe189d440599fb32a60fa430c2)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:51.732207 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54b9657-5838-48cb-b5cd-db1ea1bec551 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:51.778743 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a54b9657-5838-48cb-b5cd-db1ea1bec551 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6ebc46fe189d440599fb32a60fa430c2', 'request_id': 'req-a54b9657-5838-48cb-b5cd-db1ea1bec551'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:51.785314 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a54b9657-5838-48cb-b5cd-db1ea1bec551 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6ebc46fe189d440599fb32a60fa430c2', 'request_id': 'req-a54b9657-5838-48cb-b5cd-db1ea1bec551'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:51.791667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a54b9657-5838-48cb-b5cd-db1ea1bec551 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6ebc46fe189d440599fb32a60fa430c2', 'request_id': 'req-a54b9657-5838-48cb-b5cd-db1ea1bec551'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:51.804021 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a54b9657-5838-48cb-b5cd-db1ea1bec551 admin admin] The token cache is being invalidate because project 6ebc46fe189d440599fb32a60fa430c2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:26:51.804308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a54b9657-5838-48cb-b5cd-db1ea1bec551 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:51.804926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a54b9657-5838-48cb-b5cd-db1ea1bec551 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:51.806552 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1262/2535] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:26:51 2022] DELETE /identity/v3/projects/6ebc46fe189d440599fb32a60fa430c2 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:51.811330 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04567dd0-bdb4-428a-8580-c12d461c166a None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:51.811836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04567dd0-bdb4-428a-8580-c12d461c166a None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:51.811935 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04567dd0-bdb4-428a-8580-c12d461c166a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:51.831239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-04567dd0-bdb4-428a-8580-c12d461c166a None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:51.832411 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-04567dd0-bdb4-428a-8580-c12d461c166a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:51.890429 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1274/2536] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:51.898348 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e219608c-70a1-4e01-985c-f4c438a59ff6 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:51.898632 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e219608c-70a1-4e01-985c-f4c438a59ff6 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:51.898903 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e219608c-70a1-4e01-985c-f4c438a59ff6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:51.918793 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-e219608c-70a1-4e01-985c-f4c438a59ff6 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:51.919982 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-e219608c-70a1-4e01-985c-f4c438a59ff6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:51.928316 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1263/2537] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:51.932386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04567dd0-bdb4-428a-8580-c12d461c166a None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:51.941867 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04567dd0-bdb4-428a-8580-c12d461c166a None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:51.946554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa31e61b-3a2c-4214-a2d6-a947906efe48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:51.947145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa31e61b-3a2c-4214-a2d6-a947906efe48 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:51.947422 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa31e61b-3a2c-4214-a2d6-a947906efe48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:51.947693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa31e61b-3a2c-4214-a2d6-a947906efe48 admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:51.948324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa31e61b-3a2c-4214-a2d6-a947906efe48 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:26:51.948623 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa31e61b-3a2c-4214-a2d6-a947906efe48 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:51.950233 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa31e61b-3a2c-4214-a2d6-a947906efe48 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:51.962447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1275/2538] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:26:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:26:51.966684 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e219608c-70a1-4e01-985c-f4c438a59ff6 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:51.976634 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e219608c-70a1-4e01-985c-f4c438a59ff6 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:51.981228 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb0fe7a-bd30-49b9-939b-6f8783567215 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:51.981770 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb0fe7a-bd30-49b9-939b-6f8783567215 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:51.982065 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb0fe7a-bd30-49b9-939b-6f8783567215 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:51.982337 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb0fe7a-bd30-49b9-939b-6f8783567215 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:51.982965 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb0fe7a-bd30-49b9-939b-6f8783567215 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:51.984858 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb0fe7a-bd30-49b9-939b-6f8783567215 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:52.011014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1264/2539] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:26:51 2022] POST /identity/v3/projects => generated 379 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:26:52.015007 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa31e61b-3a2c-4214-a2d6-a947906efe48 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:52.021780 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:52.025896 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa31e61b-3a2c-4214-a2d6-a947906efe48 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:52.082601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752ff625-37f1-4bd5-9351-29ec7da7a0bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:52.083201 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752ff625-37f1-4bd5-9351-29ec7da7a0bb admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:52.083481 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752ff625-37f1-4bd5-9351-29ec7da7a0bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:52.083761 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752ff625-37f1-4bd5-9351-29ec7da7a0bb admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:52.084411 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752ff625-37f1-4bd5-9351-29ec7da7a0bb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:52.086417 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752ff625-37f1-4bd5-9351-29ec7da7a0bb admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:52.107438 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1276/2540] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:26:52 2022] POST /identity/v3/users => generated 290 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:26:52.111737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb0fe7a-bd30-49b9-939b-6f8783567215 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:52.122150 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb0fe7a-bd30-49b9-939b-6f8783567215 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:52.126766 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0c26f0-4f05-412d-abd2-474d74c8384e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:52.127362 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0c26f0-4f05-412d-abd2-474d74c8384e admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:52.127666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0c26f0-4f05-412d-abd2-474d74c8384e admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:52.127943 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0c26f0-4f05-412d-abd2-474d74c8384e admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:52.128590 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0c26f0-4f05-412d-abd2-474d74c8384e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:52.130216 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0c26f0-4f05-412d-abd2-474d74c8384e admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:52.140704 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1265/2541] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:52 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:52.145183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752ff625-37f1-4bd5-9351-29ec7da7a0bb admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:52.155316 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752ff625-37f1-4bd5-9351-29ec7da7a0bb admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:52.167412 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52cbf15-63d6-42a9-96e2-669cccd893b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:52.167412 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52cbf15-63d6-42a9-96e2-669cccd893b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:52.167412 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52cbf15-63d6-42a9-96e2-669cccd893b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:52.167412 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52cbf15-63d6-42a9-96e2-669cccd893b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:52.167801 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52cbf15-63d6-42a9-96e2-669cccd893b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:52.167801 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52cbf15-63d6-42a9-96e2-669cccd893b0 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:52.178944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1277/2542] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:52 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:52.180095 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0c26f0-4f05-412d-abd2-474d74c8384e admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:52.192408 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0c26f0-4f05-412d-abd2-474d74c8384e admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:52.197198 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4f3e44-ab9d-494f-a06a-f2fda64abfdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:52.197874 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4f3e44-ab9d-494f-a06a-f2fda64abfdc admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:52.198154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4f3e44-ab9d-494f-a06a-f2fda64abfdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:52.198426 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4f3e44-ab9d-494f-a06a-f2fda64abfdc admin admin] PATH_INFO: `/v3/projects/c56091ecf8c641bab1ac29aa633dfb07/users/9072ef47a922408bbb82262d28d579ae/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:52.213702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4f3e44-ab9d-494f-a06a-f2fda64abfdc admin admin] RBAC: Authorizing `identity:create_grant(project_id=c56091ecf8c641bab1ac29aa633dfb07, user_id=9072ef47a922408bbb82262d28d579ae, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:52.216337 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4f3e44-ab9d-494f-a06a-f2fda64abfdc admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:52.245881 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1266/2543] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:26:52 2022] PUT /identity/v3/projects/c56091ecf8c641bab1ac29aa633dfb07/users/9072ef47a922408bbb82262d28d579ae/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:52.298785 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12093985-b571-411d-9126-4e2e12eff5da None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:52.299427 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12093985-b571-411d-9126-4e2e12eff5da None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:52.299427 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12093985-b571-411d-9126-4e2e12eff5da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:52.329483 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-12093985-b571-411d-9126-4e2e12eff5da None None] MFA Rules not processed for user `9072ef47a922408bbb82262d28d579ae`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:52.330738 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-12093985-b571-411d-9126-4e2e12eff5da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:52.397906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1278/2544] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:52 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:52.402709 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4f3e44-ab9d-494f-a06a-f2fda64abfdc admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:52.440776 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:52.485913 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4f3e44-ab9d-494f-a06a-f2fda64abfdc admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:52.492371 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b9852a-b913-49e3-a55d-58d4d5ab4350 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:52.493095 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b9852a-b913-49e3-a55d-58d4d5ab4350 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:52.493564 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b9852a-b913-49e3-a55d-58d4d5ab4350 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:52.493958 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b9852a-b913-49e3-a55d-58d4d5ab4350 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:52.494922 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b9852a-b913-49e3-a55d-58d4d5ab4350 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:52.497055 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b9852a-b913-49e3-a55d-58d4d5ab4350 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:52.530997 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1267/2545] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:26:52 2022] POST /identity/v3/projects => generated 379 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:26:52.532248 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12093985-b571-411d-9126-4e2e12eff5da None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:52.552698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12093985-b571-411d-9126-4e2e12eff5da None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:52.620519 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0272946-6866-4f07-bfd5-a2d64c838374 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:52.621117 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0272946-6866-4f07-bfd5-a2d64c838374 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:52.621408 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0272946-6866-4f07-bfd5-a2d64c838374 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:52.621659 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0272946-6866-4f07-bfd5-a2d64c838374 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:52.622300 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0272946-6866-4f07-bfd5-a2d64c838374 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:52.624283 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0272946-6866-4f07-bfd5-a2d64c838374 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:52.645631 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1279/2546] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:26:52 2022] POST /identity/v3/users => generated 296 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:26:52.649820 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b9852a-b913-49e3-a55d-58d4d5ab4350 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:52.660098 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b9852a-b913-49e3-a55d-58d4d5ab4350 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:52.665554 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd9df7a-d819-4c7a-bd6f-caba3158d109 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:52.666169 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd9df7a-d819-4c7a-bd6f-caba3158d109 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:52.666446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd9df7a-d819-4c7a-bd6f-caba3158d109 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:52.666713 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd9df7a-d819-4c7a-bd6f-caba3158d109 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:52.667375 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd9df7a-d819-4c7a-bd6f-caba3158d109 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:52.668947 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd9df7a-d819-4c7a-bd6f-caba3158d109 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:52.680048 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1268/2547] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:52 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:52.684180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0272946-6866-4f07-bfd5-a2d64c838374 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:52.689982 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:52.696240 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0272946-6866-4f07-bfd5-a2d64c838374 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:52.701093 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d176c545-f11a-471f-b495-00eaa6b1ed2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:52.701748 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d176c545-f11a-471f-b495-00eaa6b1ed2f admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:52.702080 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d176c545-f11a-471f-b495-00eaa6b1ed2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:52.702385 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d176c545-f11a-471f-b495-00eaa6b1ed2f admin admin] PATH_INFO: `/v3/projects/fd11e5581c884623827d5b683c0ac58a/users/7a9784c5641d4f8999f5e3f46359f24f/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:52.716727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d176c545-f11a-471f-b495-00eaa6b1ed2f admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd11e5581c884623827d5b683c0ac58a, user_id=7a9784c5641d4f8999f5e3f46359f24f, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:52.719433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d176c545-f11a-471f-b495-00eaa6b1ed2f admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:52.732822 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1280/2548] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:26:52 2022] PUT /identity/v3/projects/fd11e5581c884623827d5b683c0ac58a/users/7a9784c5641d4f8999f5e3f46359f24f/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:52.737235 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd9df7a-d819-4c7a-bd6f-caba3158d109 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:52.747627 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd9df7a-d819-4c7a-bd6f-caba3158d109 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:52.804839 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c2aee4-27ee-41b6-bbf0-972020ffe824 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:52.805433 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c2aee4-27ee-41b6-bbf0-972020ffe824 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:52.805713 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c2aee4-27ee-41b6-bbf0-972020ffe824 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:52.806021 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c2aee4-27ee-41b6-bbf0-972020ffe824 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:52.806654 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11c2aee4-27ee-41b6-bbf0-972020ffe824 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:52.808260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11c2aee4-27ee-41b6-bbf0-972020ffe824 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:52.818502 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1269/2549] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:52 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:52.822721 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d176c545-f11a-471f-b495-00eaa6b1ed2f admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:52.833018 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d176c545-f11a-471f-b495-00eaa6b1ed2f admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:52.837562 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b103745f-9ac1-4d13-a05c-e213e962bf0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:52.838235 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b103745f-9ac1-4d13-a05c-e213e962bf0c admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:52.838513 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b103745f-9ac1-4d13-a05c-e213e962bf0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:52.838784 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b103745f-9ac1-4d13-a05c-e213e962bf0c admin admin] PATH_INFO: `/v3/projects/fd11e5581c884623827d5b683c0ac58a/users/7a9784c5641d4f8999f5e3f46359f24f/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:52.840651 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b103745f-9ac1-4d13-a05c-e213e962bf0c admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd11e5581c884623827d5b683c0ac58a, user_id=7a9784c5641d4f8999f5e3f46359f24f, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:52.843300 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b103745f-9ac1-4d13-a05c-e213e962bf0c admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:52.853279 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1281/2550] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:26:52 2022] PUT /identity/v3/projects/fd11e5581c884623827d5b683c0ac58a/users/7a9784c5641d4f8999f5e3f46359f24f/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:52.863877 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a17aaf-0d4b-4796-b71f-3d52c46fbfd0 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:52.864165 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a17aaf-0d4b-4796-b71f-3d52c46fbfd0 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:52.864442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a17aaf-0d4b-4796-b71f-3d52c46fbfd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:52.892241 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-a5a17aaf-0d4b-4796-b71f-3d52c46fbfd0 None None] MFA Rules not processed for user `7a9784c5641d4f8999f5e3f46359f24f`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:52.893446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-a5a17aaf-0d4b-4796-b71f-3d52c46fbfd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:52.959131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1270/2551] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:52 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:26:52.971418 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b103745f-9ac1-4d13-a05c-e213e962bf0c admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:52.973077 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-b103745f-9ac1-4d13-a05c-e213e962bf0c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:53.028933 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b103745f-9ac1-4d13-a05c-e213e962bf0c admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:53.032476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fa8ff6-d835-49f3-8f1b-b84230082d2c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:53.033072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fa8ff6-d835-49f3-8f1b-b84230082d2c service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:53.033351 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fa8ff6-d835-49f3-8f1b-b84230082d2c service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:53.033624 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fa8ff6-d835-49f3-8f1b-b84230082d2c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:53.042657 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5fa8ff6-d835-49f3-8f1b-b84230082d2c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:53.044075 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5fa8ff6-d835-49f3-8f1b-b84230082d2c service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:53.053692 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1282/2552] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:26:52 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:26:53.364409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a17aaf-0d4b-4796-b71f-3d52c46fbfd0 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:53.373582 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a17aaf-0d4b-4796-b71f-3d52c46fbfd0 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:53.378717 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a15d2f-d488-47f3-a117-4026ed22845d tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7a9784c5641d4f8999f5e3f46359f24f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd11e5581c884623827d5b683c0ac58a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:53.379313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a15d2f-d488-47f3-a117-4026ed22845d tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:53.379596 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a15d2f-d488-47f3-a117-4026ed22845d tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:53.379880 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a15d2f-d488-47f3-a117-4026ed22845d tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:53.380559 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a15d2f-d488-47f3-a117-4026ed22845d tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:26:53.380865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a15d2f-d488-47f3-a117-4026ed22845d tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:53.382496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a15d2f-d488-47f3-a117-4026ed22845d tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:53.394464 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1271/2553] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:26:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:26:53.398390 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fa8ff6-d835-49f3-8f1b-b84230082d2c service cinder] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:53.407361 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fa8ff6-d835-49f3-8f1b-b84230082d2c service cinder] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:53.411875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cde36f5-42b8-476d-ba11-88a2e9497ee3 tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7a9784c5641d4f8999f5e3f46359f24f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd11e5581c884623827d5b683c0ac58a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:53.412464 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cde36f5-42b8-476d-ba11-88a2e9497ee3 tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:53.412742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cde36f5-42b8-476d-ba11-88a2e9497ee3 tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:53.413014 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cde36f5-42b8-476d-ba11-88a2e9497ee3 tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:53.413649 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cde36f5-42b8-476d-ba11-88a2e9497ee3 tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:53.415836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cde36f5-42b8-476d-ba11-88a2e9497ee3 tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:53.442806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1283/2554] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:26:53 2022] POST /identity/v3/projects => generated 353 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:26:53.446879 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a15d2f-d488-47f3-a117-4026ed22845d tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:53.456916 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a15d2f-d488-47f3-a117-4026ed22845d tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:53.468137 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:53.513753 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57b11fd-2957-4aa9-8934-2be079195ac5 tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7a9784c5641d4f8999f5e3f46359f24f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd11e5581c884623827d5b683c0ac58a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:53.514368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57b11fd-2957-4aa9-8934-2be079195ac5 tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:53.514652 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57b11fd-2957-4aa9-8934-2be079195ac5 tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:53.514929 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57b11fd-2957-4aa9-8934-2be079195ac5 tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:53.515573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57b11fd-2957-4aa9-8934-2be079195ac5 tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:26:53.515878 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57b11fd-2957-4aa9-8934-2be079195ac5 tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:53.517491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57b11fd-2957-4aa9-8934-2be079195ac5 tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:53.530147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1272/2555] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:26:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:26:53.569629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cde36f5-42b8-476d-ba11-88a2e9497ee3 tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:53.579956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cde36f5-42b8-476d-ba11-88a2e9497ee3 tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:53.584743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ca7781-193c-4359-9608-3e2ba904279c tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7a9784c5641d4f8999f5e3f46359f24f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd11e5581c884623827d5b683c0ac58a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:53.585363 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ca7781-193c-4359-9608-3e2ba904279c tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:53.585641 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ca7781-193c-4359-9608-3e2ba904279c tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:53.585945 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ca7781-193c-4359-9608-3e2ba904279c tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] PATH_INFO: `/v3/projects/43901dbba4e24dbdaf5088c9e03b5f9e` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:53.587097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ca7781-193c-4359-9608-3e2ba904279c tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] RBAC: Authorizing `identity:delete_project(project_id=43901dbba4e24dbdaf5088c9e03b5f9e)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:53.588989 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ca7781-193c-4359-9608-3e2ba904279c tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:53.641971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-69ca7781-193c-4359-9608-3e2ba904279c tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '43901dbba4e24dbdaf5088c9e03b5f9e', 'request_id': 'req-69ca7781-193c-4359-9608-3e2ba904279c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:53.642176 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-69ca7781-193c-4359-9608-3e2ba904279c tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '43901dbba4e24dbdaf5088c9e03b5f9e', 'request_id': 'req-69ca7781-193c-4359-9608-3e2ba904279c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:53.649441 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-69ca7781-193c-4359-9608-3e2ba904279c tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '43901dbba4e24dbdaf5088c9e03b5f9e', 'request_id': 'req-69ca7781-193c-4359-9608-3e2ba904279c'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:53.662761 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-69ca7781-193c-4359-9608-3e2ba904279c tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] The token cache is being invalidate because project 43901dbba4e24dbdaf5088c9e03b5f9e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:26:53.663045 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-69ca7781-193c-4359-9608-3e2ba904279c tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:53.663673 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-69ca7781-193c-4359-9608-3e2ba904279c tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:53.665413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1284/2556] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:26:53 2022] DELETE /identity/v3/projects/43901dbba4e24dbdaf5088c9e03b5f9e => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:53.874870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57b11fd-2957-4aa9-8934-2be079195ac5 tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:53.876625 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c57b11fd-2957-4aa9-8934-2be079195ac5 tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:53.939695 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57b11fd-2957-4aa9-8934-2be079195ac5 tempest-VolumeQuotaClassesTest-642034239 tempest-VolumeQuotaClassesTest-642034239-project-admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:53.995015 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5f0d21-6899-41e5-8557-cca59620acfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:53.995640 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5f0d21-6899-41e5-8557-cca59620acfe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:53.995969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5f0d21-6899-41e5-8557-cca59620acfe admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:53.996238 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5f0d21-6899-41e5-8557-cca59620acfe admin admin] PATH_INFO: `/v3/users/9072ef47a922408bbb82262d28d579ae` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:53.997390 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d5f0d21-6899-41e5-8557-cca59620acfe admin admin] RBAC: Authorizing `identity:delete_user(user_id=9072ef47a922408bbb82262d28d579ae)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:53.999355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d5f0d21-6899-41e5-8557-cca59620acfe admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:54.044341 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-6d5f0d21-6899-41e5-8557-cca59620acfe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9072ef47a922408bbb82262d28d579ae'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:54.051849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-6d5f0d21-6899-41e5-8557-cca59620acfe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9072ef47a922408bbb82262d28d579ae'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:54.052498 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-6d5f0d21-6899-41e5-8557-cca59620acfe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9072ef47a922408bbb82262d28d579ae'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:54.054996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:54.062190 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-6d5f0d21-6899-41e5-8557-cca59620acfe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9072ef47a922408bbb82262d28d579ae'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:54.076370 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-6d5f0d21-6899-41e5-8557-cca59620acfe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9072ef47a922408bbb82262d28d579ae'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:54.082315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1273/2557] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:26:53 2022] DELETE /identity/v3/users/9072ef47a922408bbb82262d28d579ae => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:54.086858 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ab969b-274f-49c6-9422-90a337e0df97 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:54.087301 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ab969b-274f-49c6-9422-90a337e0df97 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:54.087423 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ab969b-274f-49c6-9422-90a337e0df97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:54.106255 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-46ab969b-274f-49c6-9422-90a337e0df97 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:54.107449 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-46ab969b-274f-49c6-9422-90a337e0df97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:54.130945 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:54.166791 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1285/2558] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:54.177111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5f0d21-6899-41e5-8557-cca59620acfe admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:54.178543 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-6d5f0d21-6899-41e5-8557-cca59620acfe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:54.236952 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5f0d21-6899-41e5-8557-cca59620acfe admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:54.240547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a292343c-4321-4505-8850-1cd314bb0a6e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:54.241151 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a292343c-4321-4505-8850-1cd314bb0a6e service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:54.241425 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a292343c-4321-4505-8850-1cd314bb0a6e service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:54.241695 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a292343c-4321-4505-8850-1cd314bb0a6e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:54.250554 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a292343c-4321-4505-8850-1cd314bb0a6e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:54.251964 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a292343c-4321-4505-8850-1cd314bb0a6e service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:54.261122 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1274/2559] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:26:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:26:54.775648 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ab969b-274f-49c6-9422-90a337e0df97 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:54.777113 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-46ab969b-274f-49c6-9422-90a337e0df97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:54.791240 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ab969b-274f-49c6-9422-90a337e0df97 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:54.796894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab7afb8-1432-43b3-be83-459a1f43d682 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:54.797823 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab7afb8-1432-43b3-be83-459a1f43d682 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:54.798313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab7afb8-1432-43b3-be83-459a1f43d682 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:54.798640 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab7afb8-1432-43b3-be83-459a1f43d682 admin admin] PATH_INFO: `/v3/projects/c56091ecf8c641bab1ac29aa633dfb07` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:54.799825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab7afb8-1432-43b3-be83-459a1f43d682 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c56091ecf8c641bab1ac29aa633dfb07)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:54.801819 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab7afb8-1432-43b3-be83-459a1f43d682 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:54.848380 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-eab7afb8-1432-43b3-be83-459a1f43d682 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c56091ecf8c641bab1ac29aa633dfb07', 'request_id': 'req-eab7afb8-1432-43b3-be83-459a1f43d682'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:54.854686 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-eab7afb8-1432-43b3-be83-459a1f43d682 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c56091ecf8c641bab1ac29aa633dfb07', 'request_id': 'req-eab7afb8-1432-43b3-be83-459a1f43d682'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:54.862118 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-eab7afb8-1432-43b3-be83-459a1f43d682 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c56091ecf8c641bab1ac29aa633dfb07', 'request_id': 'req-eab7afb8-1432-43b3-be83-459a1f43d682'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:54.877519 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-eab7afb8-1432-43b3-be83-459a1f43d682 admin admin] The token cache is being invalidate because project c56091ecf8c641bab1ac29aa633dfb07 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:26:54.877852 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-eab7afb8-1432-43b3-be83-459a1f43d682 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:54.878476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-eab7afb8-1432-43b3-be83-459a1f43d682 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:54.879986 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1286/2560] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:26:54 2022] DELETE /identity/v3/projects/c56091ecf8c641bab1ac29aa633dfb07 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:54.883906 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a292343c-4321-4505-8850-1cd314bb0a6e service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:54.894005 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a292343c-4321-4505-8850-1cd314bb0a6e service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:54.949262 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e767adb-37f9-4f8d-b6f4-9e0d2d245574 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:54.949905 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e767adb-37f9-4f8d-b6f4-9e0d2d245574 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:54.950185 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e767adb-37f9-4f8d-b6f4-9e0d2d245574 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:54.950511 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e767adb-37f9-4f8d-b6f4-9e0d2d245574 admin admin] PATH_INFO: `/v3/users/7a9784c5641d4f8999f5e3f46359f24f` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:54.951602 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e767adb-37f9-4f8d-b6f4-9e0d2d245574 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7a9784c5641d4f8999f5e3f46359f24f)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:54.953487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e767adb-37f9-4f8d-b6f4-9e0d2d245574 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:54.997271 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-5e767adb-37f9-4f8d-b6f4-9e0d2d245574 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7a9784c5641d4f8999f5e3f46359f24f'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:55.006376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-5e767adb-37f9-4f8d-b6f4-9e0d2d245574 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7a9784c5641d4f8999f5e3f46359f24f'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:55.007043 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-5e767adb-37f9-4f8d-b6f4-9e0d2d245574 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7a9784c5641d4f8999f5e3f46359f24f'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:55.014387 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-5e767adb-37f9-4f8d-b6f4-9e0d2d245574 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7a9784c5641d4f8999f5e3f46359f24f'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:55.026035 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-5e767adb-37f9-4f8d-b6f4-9e0d2d245574 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7a9784c5641d4f8999f5e3f46359f24f'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:55.027370 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:55.033028 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1275/2561] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:26:54 2022] DELETE /identity/v3/users/7a9784c5641d4f8999f5e3f46359f24f => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:55.307182 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab7afb8-1432-43b3-be83-459a1f43d682 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:55.308772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-eab7afb8-1432-43b3-be83-459a1f43d682 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:55.324015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab7afb8-1432-43b3-be83-459a1f43d682 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:55.382455 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940fbdb2-de1b-4912-b620-a054ec28c46f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:55.383105 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940fbdb2-de1b-4912-b620-a054ec28c46f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:55.383376 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940fbdb2-de1b-4912-b620-a054ec28c46f admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:55.383643 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940fbdb2-de1b-4912-b620-a054ec28c46f admin admin] PATH_INFO: `/v3/projects/fd11e5581c884623827d5b683c0ac58a` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:55.384728 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940fbdb2-de1b-4912-b620-a054ec28c46f admin admin] RBAC: Authorizing `identity:delete_project(project_id=fd11e5581c884623827d5b683c0ac58a)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:55.386627 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940fbdb2-de1b-4912-b620-a054ec28c46f admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:55.433693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-940fbdb2-de1b-4912-b620-a054ec28c46f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fd11e5581c884623827d5b683c0ac58a', 'request_id': 'req-940fbdb2-de1b-4912-b620-a054ec28c46f'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:55.437512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:55.440392 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-940fbdb2-de1b-4912-b620-a054ec28c46f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fd11e5581c884623827d5b683c0ac58a', 'request_id': 'req-940fbdb2-de1b-4912-b620-a054ec28c46f'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:55.447942 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-940fbdb2-de1b-4912-b620-a054ec28c46f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fd11e5581c884623827d5b683c0ac58a', 'request_id': 'req-940fbdb2-de1b-4912-b620-a054ec28c46f'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:55.460689 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-940fbdb2-de1b-4912-b620-a054ec28c46f admin admin] The token cache is being invalidate because project fd11e5581c884623827d5b683c0ac58a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:26:55.460984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-940fbdb2-de1b-4912-b620-a054ec28c46f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:55.461614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-940fbdb2-de1b-4912-b620-a054ec28c46f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:55.463220 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1287/2562] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:26:55 2022] DELETE /identity/v3/projects/fd11e5581c884623827d5b683c0ac58a => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:55.468074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef16990a-0257-4f7a-828e-e697a09a78b0 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:55.468359 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef16990a-0257-4f7a-828e-e697a09a78b0 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:55.468630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef16990a-0257-4f7a-828e-e697a09a78b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:55.488756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-ef16990a-0257-4f7a-828e-e697a09a78b0 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:55.489911 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ef16990a-0257-4f7a-828e-e697a09a78b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:55.547038 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1276/2563] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:55.555321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe491ec-f0de-42f3-a45f-a9541766116d None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:55.555600 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe491ec-f0de-42f3-a45f-a9541766116d None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:55.555873 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe491ec-f0de-42f3-a45f-a9541766116d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:55.575118 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-bfe491ec-f0de-42f3-a45f-a9541766116d None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:55.576295 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-bfe491ec-f0de-42f3-a45f-a9541766116d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:55.583995 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1288/2564] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:26:55.588019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef16990a-0257-4f7a-828e-e697a09a78b0 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:55.598807 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef16990a-0257-4f7a-828e-e697a09a78b0 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:55.603431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f27f32a-ad03-46cf-8561-a6e915f975f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:55.604019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f27f32a-ad03-46cf-8561-a6e915f975f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:55.604301 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f27f32a-ad03-46cf-8561-a6e915f975f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:55.604574 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f27f32a-ad03-46cf-8561-a6e915f975f7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:55.606993 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f27f32a-ad03-46cf-8561-a6e915f975f7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:26:55.606993 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f27f32a-ad03-46cf-8561-a6e915f975f7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:55.607221 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f27f32a-ad03-46cf-8561-a6e915f975f7 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:55.619579 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1277/2565] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:26:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:26:55.623812 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe491ec-f0de-42f3-a45f-a9541766116d None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:55.632896 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe491ec-f0de-42f3-a45f-a9541766116d None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:55.637541 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9dc5a6-9f1d-4348-a73e-9613fe5a77fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:55.638224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9dc5a6-9f1d-4348-a73e-9613fe5a77fe admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:55.638562 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9dc5a6-9f1d-4348-a73e-9613fe5a77fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:55.638892 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9dc5a6-9f1d-4348-a73e-9613fe5a77fe admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:55.639586 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9dc5a6-9f1d-4348-a73e-9613fe5a77fe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:55.641567 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9dc5a6-9f1d-4348-a73e-9613fe5a77fe admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:55.673607 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1289/2566] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:26:55 2022] POST /identity/v3/projects => generated 389 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:26:55.674791 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f27f32a-ad03-46cf-8561-a6e915f975f7 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:55.685110 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f27f32a-ad03-46cf-8561-a6e915f975f7 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:55.740546 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c882ef-ab75-4a62-a8ec-132faaf6f750 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:55.741079 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c882ef-ab75-4a62-a8ec-132faaf6f750 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:55.741356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c882ef-ab75-4a62-a8ec-132faaf6f750 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:55.741623 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c882ef-ab75-4a62-a8ec-132faaf6f750 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:55.742267 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c882ef-ab75-4a62-a8ec-132faaf6f750 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:55.744156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c882ef-ab75-4a62-a8ec-132faaf6f750 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:55.764802 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1278/2567] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:26:55 2022] POST /identity/v3/users => generated 295 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:26:55.769396 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9dc5a6-9f1d-4348-a73e-9613fe5a77fe admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:55.779500 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9dc5a6-9f1d-4348-a73e-9613fe5a77fe admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:55.784593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd57fff-2fca-4df4-be93-968c43d0e34f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:55.785170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd57fff-2fca-4df4-be93-968c43d0e34f admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:55.785442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd57fff-2fca-4df4-be93-968c43d0e34f admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:55.785711 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd57fff-2fca-4df4-be93-968c43d0e34f admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:55.786367 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd57fff-2fca-4df4-be93-968c43d0e34f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:55.787932 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd57fff-2fca-4df4-be93-968c43d0e34f admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:55.798329 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1290/2568] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:55 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:55.802129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c882ef-ab75-4a62-a8ec-132faaf6f750 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:55.811855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c882ef-ab75-4a62-a8ec-132faaf6f750 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:55.816556 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a522011b-c441-450c-98a2-0bf4a654954f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:55.817143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a522011b-c441-450c-98a2-0bf4a654954f admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:55.817423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a522011b-c441-450c-98a2-0bf4a654954f admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:55.817697 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a522011b-c441-450c-98a2-0bf4a654954f admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:55.818378 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a522011b-c441-450c-98a2-0bf4a654954f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:55.819944 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a522011b-c441-450c-98a2-0bf4a654954f admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:55.830942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1279/2569] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:55 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:55.834744 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd57fff-2fca-4df4-be93-968c43d0e34f admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:55.845173 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd57fff-2fca-4df4-be93-968c43d0e34f admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:55.849711 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697f4807-eb60-45f8-b0f4-b2574ef93c16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:55.850827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697f4807-eb60-45f8-b0f4-b2574ef93c16 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:55.851458 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697f4807-eb60-45f8-b0f4-b2574ef93c16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:55.851860 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697f4807-eb60-45f8-b0f4-b2574ef93c16 admin admin] PATH_INFO: `/v3/projects/b718062f86b14cc7b1ce38b8c46567bd/users/2a1aac260e184ba8bdbe800a1f198a67/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:55.866775 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697f4807-eb60-45f8-b0f4-b2574ef93c16 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b718062f86b14cc7b1ce38b8c46567bd, user_id=2a1aac260e184ba8bdbe800a1f198a67, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:55.869400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697f4807-eb60-45f8-b0f4-b2574ef93c16 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:55.882935 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1291/2570] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:26:55 2022] PUT /identity/v3/projects/b718062f86b14cc7b1ce38b8c46567bd/users/2a1aac260e184ba8bdbe800a1f198a67/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:26:55.895991 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fb92f4-32b5-48aa-89bd-1a4c1dfc8397 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:55.895991 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fb92f4-32b5-48aa-89bd-1a4c1dfc8397 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:55.895991 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fb92f4-32b5-48aa-89bd-1a4c1dfc8397 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:55.921852 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-21fb92f4-32b5-48aa-89bd-1a4c1dfc8397 None None] MFA Rules not processed for user `2a1aac260e184ba8bdbe800a1f198a67`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:55.923074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-21fb92f4-32b5-48aa-89bd-1a4c1dfc8397 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:55.977667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:55.989016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1280/2571] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:55 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:55.993776 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697f4807-eb60-45f8-b0f4-b2574ef93c16 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:56.054071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697f4807-eb60-45f8-b0f4-b2574ef93c16 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:56.058702 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7229c22-9f01-4ea0-b6dd-87b0e4fa9e47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:56.059300 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7229c22-9f01-4ea0-b6dd-87b0e4fa9e47 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:56.059579 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7229c22-9f01-4ea0-b6dd-87b0e4fa9e47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:56.059852 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7229c22-9f01-4ea0-b6dd-87b0e4fa9e47 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:56.060469 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7229c22-9f01-4ea0-b6dd-87b0e4fa9e47 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:56.062421 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7229c22-9f01-4ea0-b6dd-87b0e4fa9e47 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:56.088822 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1292/2572] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:26:55 2022] POST /identity/v3/projects => generated 387 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:26:56.092834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fb92f4-32b5-48aa-89bd-1a4c1dfc8397 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:56.103268 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fb92f4-32b5-48aa-89bd-1a4c1dfc8397 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:56.158181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f072a0-dc0c-4391-9df7-5082978bdfac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:56.158789 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f072a0-dc0c-4391-9df7-5082978bdfac admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:56.159066 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f072a0-dc0c-4391-9df7-5082978bdfac admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:56.159339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f072a0-dc0c-4391-9df7-5082978bdfac admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:56.159965 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f072a0-dc0c-4391-9df7-5082978bdfac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:56.161887 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f072a0-dc0c-4391-9df7-5082978bdfac admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:56.182552 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1281/2573] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:26:56 2022] POST /identity/v3/users => generated 300 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:26:56.187241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7229c22-9f01-4ea0-b6dd-87b0e4fa9e47 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:56.197906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7229c22-9f01-4ea0-b6dd-87b0e4fa9e47 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:56.202495 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602b02d5-a300-45df-9c5a-cd6eee18f4c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:56.203085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602b02d5-a300-45df-9c5a-cd6eee18f4c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:56.203362 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602b02d5-a300-45df-9c5a-cd6eee18f4c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:56.203634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602b02d5-a300-45df-9c5a-cd6eee18f4c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:56.204806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602b02d5-a300-45df-9c5a-cd6eee18f4c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:56.206411 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602b02d5-a300-45df-9c5a-cd6eee18f4c1 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:56.216864 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1293/2574] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:56 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:56.220967 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f072a0-dc0c-4391-9df7-5082978bdfac admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:56.231109 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f072a0-dc0c-4391-9df7-5082978bdfac admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:56.235665 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598f7fa7-969d-4f49-8e7d-519262ea6a77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:56.236334 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598f7fa7-969d-4f49-8e7d-519262ea6a77 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:56.236615 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598f7fa7-969d-4f49-8e7d-519262ea6a77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:56.236888 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598f7fa7-969d-4f49-8e7d-519262ea6a77 admin admin] PATH_INFO: `/v3/projects/c0d1e5a9ecb84d10883a49e351a5ad4f/users/085afc09c5e04a3dbed4243b58c998a4/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:56.251716 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598f7fa7-969d-4f49-8e7d-519262ea6a77 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c0d1e5a9ecb84d10883a49e351a5ad4f, user_id=085afc09c5e04a3dbed4243b58c998a4, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:56.254833 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598f7fa7-969d-4f49-8e7d-519262ea6a77 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:56.267481 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1282/2575] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:26:56 2022] PUT /identity/v3/projects/c0d1e5a9ecb84d10883a49e351a5ad4f/users/085afc09c5e04a3dbed4243b58c998a4/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:56.271444 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602b02d5-a300-45df-9c5a-cd6eee18f4c1 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:56.281069 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602b02d5-a300-45df-9c5a-cd6eee18f4c1 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:56.326982 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:DELETE=2,keystone:SELECT=94,keystone:INSERT=4 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:56.338202 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de20b9ef-3aa9-417c-9c68-55752de2668d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:56.338976 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de20b9ef-3aa9-417c-9c68-55752de2668d admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:56.339316 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de20b9ef-3aa9-417c-9c68-55752de2668d admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:56.339830 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de20b9ef-3aa9-417c-9c68-55752de2668d admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:56.340505 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de20b9ef-3aa9-417c-9c68-55752de2668d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:56.342145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de20b9ef-3aa9-417c-9c68-55752de2668d admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:56.351822 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1294/2576] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:56 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:56.356329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598f7fa7-969d-4f49-8e7d-519262ea6a77 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:56.366735 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598f7fa7-969d-4f49-8e7d-519262ea6a77 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:56.371306 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8c75aa-850a-404a-8ebb-226a398ffc6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:56.371952 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8c75aa-850a-404a-8ebb-226a398ffc6f admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:56.372236 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8c75aa-850a-404a-8ebb-226a398ffc6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:56.372516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8c75aa-850a-404a-8ebb-226a398ffc6f admin admin] PATH_INFO: `/v3/projects/c0d1e5a9ecb84d10883a49e351a5ad4f/users/085afc09c5e04a3dbed4243b58c998a4/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:56.374419 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8c75aa-850a-404a-8ebb-226a398ffc6f admin admin] RBAC: Authorizing `identity:create_grant(project_id=c0d1e5a9ecb84d10883a49e351a5ad4f, user_id=085afc09c5e04a3dbed4243b58c998a4, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:56.377519 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8c75aa-850a-404a-8ebb-226a398ffc6f admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:56.387547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1283/2577] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:26:56 2022] PUT /identity/v3/projects/c0d1e5a9ecb84d10883a49e351a5ad4f/users/085afc09c5e04a3dbed4243b58c998a4/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:56.398245 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b15dae-f974-4e26-bb53-948bed27cc4e None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:56.398594 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b15dae-f974-4e26-bb53-948bed27cc4e None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:56.398871 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b15dae-f974-4e26-bb53-948bed27cc4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:56.427625 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-70b15dae-f974-4e26-bb53-948bed27cc4e None None] MFA Rules not processed for user `085afc09c5e04a3dbed4243b58c998a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:56.428810 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-70b15dae-f974-4e26-bb53-948bed27cc4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:56.494142 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1295/2578] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:56 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:56.505395 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8c75aa-850a-404a-8ebb-226a398ffc6f admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:56.507068 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-2a8c75aa-850a-404a-8ebb-226a398ffc6f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:56.566618 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8c75aa-850a-404a-8ebb-226a398ffc6f admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:56.570165 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896cba57-ef23-4a26-a546-b4f01f09af63 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:56.570761 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896cba57-ef23-4a26-a546-b4f01f09af63 service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:56.571036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896cba57-ef23-4a26-a546-b4f01f09af63 service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:56.571307 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896cba57-ef23-4a26-a546-b4f01f09af63 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:56.580158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896cba57-ef23-4a26-a546-b4f01f09af63 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:56.581598 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896cba57-ef23-4a26-a546-b4f01f09af63 service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:56.590846 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1284/2579] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:26:56 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:26:56.726271 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b15dae-f974-4e26-bb53-948bed27cc4e None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:56.787274 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b15dae-f974-4e26-bb53-948bed27cc4e None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:56.792076 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5de0a0-d52d-49c2-a9c2-e0ac4f86c281 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:56.792700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5de0a0-d52d-49c2-a9c2-e0ac4f86c281 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:56.792976 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5de0a0-d52d-49c2-a9c2-e0ac4f86c281 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:56.793247 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5de0a0-d52d-49c2-a9c2-e0ac4f86c281 admin admin] PATH_INFO: `/v3/users/2a1aac260e184ba8bdbe800a1f198a67` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:56.794397 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5de0a0-d52d-49c2-a9c2-e0ac4f86c281 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2a1aac260e184ba8bdbe800a1f198a67)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:56.796381 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5de0a0-d52d-49c2-a9c2-e0ac4f86c281 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:56.846662 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0a5de0a0-d52d-49c2-a9c2-e0ac4f86c281 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2a1aac260e184ba8bdbe800a1f198a67'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:56.853425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0a5de0a0-d52d-49c2-a9c2-e0ac4f86c281 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2a1aac260e184ba8bdbe800a1f198a67'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:56.854096 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0a5de0a0-d52d-49c2-a9c2-e0ac4f86c281 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2a1aac260e184ba8bdbe800a1f198a67'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:56.862298 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0a5de0a0-d52d-49c2-a9c2-e0ac4f86c281 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2a1aac260e184ba8bdbe800a1f198a67'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:56.872581 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:56.875041 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0a5de0a0-d52d-49c2-a9c2-e0ac4f86c281 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2a1aac260e184ba8bdbe800a1f198a67'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:56.882266 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1296/2580] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:26:56 2022] DELETE /identity/v3/users/2a1aac260e184ba8bdbe800a1f198a67 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:56.886727 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea04006f-9096-4809-a3cc-7d374d955cbe None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:56.887038 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea04006f-9096-4809-a3cc-7d374d955cbe None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:56.887310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea04006f-9096-4809-a3cc-7d374d955cbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:56.906772 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-ea04006f-9096-4809-a3cc-7d374d955cbe None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:56.908529 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ea04006f-9096-4809-a3cc-7d374d955cbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:56.961893 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:56.978169 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1285/2581] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:56.988460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5de0a0-d52d-49c2-a9c2-e0ac4f86c281 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:56.989872 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-0a5de0a0-d52d-49c2-a9c2-e0ac4f86c281 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:57.046115 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5de0a0-d52d-49c2-a9c2-e0ac4f86c281 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:57.050280 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cea491e-a002-4d92-976d-574d29b0d12a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:57.050886 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cea491e-a002-4d92-976d-574d29b0d12a service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:57.051164 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cea491e-a002-4d92-976d-574d29b0d12a service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:57.051437 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cea491e-a002-4d92-976d-574d29b0d12a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:57.060174 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cea491e-a002-4d92-976d-574d29b0d12a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:57.061585 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cea491e-a002-4d92-976d-574d29b0d12a service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:57.070892 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1297/2582] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:26:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:26:57.340706 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea04006f-9096-4809-a3cc-7d374d955cbe None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:57.342199 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ea04006f-9096-4809-a3cc-7d374d955cbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:57.355711 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea04006f-9096-4809-a3cc-7d374d955cbe None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:57.360674 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b821b4-4264-45e6-b609-62ec093d45e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:57.361343 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b821b4-4264-45e6-b609-62ec093d45e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:57.361720 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b821b4-4264-45e6-b609-62ec093d45e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:57.361948 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b821b4-4264-45e6-b609-62ec093d45e2 admin admin] PATH_INFO: `/v3/projects/b718062f86b14cc7b1ce38b8c46567bd` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:57.363608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b821b4-4264-45e6-b609-62ec093d45e2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b718062f86b14cc7b1ce38b8c46567bd)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:57.365719 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b821b4-4264-45e6-b609-62ec093d45e2 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:57.413018 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a0b821b4-4264-45e6-b609-62ec093d45e2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b718062f86b14cc7b1ce38b8c46567bd', 'request_id': 'req-a0b821b4-4264-45e6-b609-62ec093d45e2'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:57.418970 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a0b821b4-4264-45e6-b609-62ec093d45e2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b718062f86b14cc7b1ce38b8c46567bd', 'request_id': 'req-a0b821b4-4264-45e6-b609-62ec093d45e2'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:57.425458 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a0b821b4-4264-45e6-b609-62ec093d45e2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b718062f86b14cc7b1ce38b8c46567bd', 'request_id': 'req-a0b821b4-4264-45e6-b609-62ec093d45e2'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:57.437617 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a0b821b4-4264-45e6-b609-62ec093d45e2 admin admin] The token cache is being invalidate because project b718062f86b14cc7b1ce38b8c46567bd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:26:57.437938 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a0b821b4-4264-45e6-b609-62ec093d45e2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:57.438576 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a0b821b4-4264-45e6-b609-62ec093d45e2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:57.440050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1286/2583] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:26:57 2022] DELETE /identity/v3/projects/b718062f86b14cc7b1ce38b8c46567bd => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:57.444070 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cea491e-a002-4d92-976d-574d29b0d12a service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:57.453417 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cea491e-a002-4d92-976d-574d29b0d12a service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:57.508689 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d357df-2acf-4bec-bec7-5b68da7872fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:57.509306 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d357df-2acf-4bec-bec7-5b68da7872fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:57.509576 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d357df-2acf-4bec-bec7-5b68da7872fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:57.509879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d357df-2acf-4bec-bec7-5b68da7872fa admin admin] PATH_INFO: `/v3/users/085afc09c5e04a3dbed4243b58c998a4` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:57.510967 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d357df-2acf-4bec-bec7-5b68da7872fa admin admin] RBAC: Authorizing `identity:delete_user(user_id=085afc09c5e04a3dbed4243b58c998a4)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:57.512874 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d357df-2acf-4bec-bec7-5b68da7872fa admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:57.556798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-93d357df-2acf-4bec-bec7-5b68da7872fa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '085afc09c5e04a3dbed4243b58c998a4'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:57.563944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-93d357df-2acf-4bec-bec7-5b68da7872fa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '085afc09c5e04a3dbed4243b58c998a4'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:57.564594 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-93d357df-2acf-4bec-bec7-5b68da7872fa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '085afc09c5e04a3dbed4243b58c998a4'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:57.571273 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-93d357df-2acf-4bec-bec7-5b68da7872fa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '085afc09c5e04a3dbed4243b58c998a4'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:57.583150 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-93d357df-2acf-4bec-bec7-5b68da7872fa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '085afc09c5e04a3dbed4243b58c998a4'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:57.588893 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1298/2584] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:26:57 2022] DELETE /identity/v3/users/085afc09c5e04a3dbed4243b58c998a4 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:57.855912 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b821b4-4264-45e6-b609-62ec093d45e2 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:57.857475 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-a0b821b4-4264-45e6-b609-62ec093d45e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:57.873927 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b821b4-4264-45e6-b609-62ec093d45e2 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:57.931902 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba448e07-f84c-4285-8748-b551ca1e8c6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:57.932533 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba448e07-f84c-4285-8748-b551ca1e8c6e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:57.933036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba448e07-f84c-4285-8748-b551ca1e8c6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:57.933036 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba448e07-f84c-4285-8748-b551ca1e8c6e admin admin] PATH_INFO: `/v3/projects/c0d1e5a9ecb84d10883a49e351a5ad4f` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:57.934271 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba448e07-f84c-4285-8748-b551ca1e8c6e admin admin] RBAC: Authorizing `identity:delete_project(project_id=c0d1e5a9ecb84d10883a49e351a5ad4f)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:57.936201 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba448e07-f84c-4285-8748-b551ca1e8c6e admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:57.986430 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-ba448e07-f84c-4285-8748-b551ca1e8c6e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c0d1e5a9ecb84d10883a49e351a5ad4f', 'request_id': 'req-ba448e07-f84c-4285-8748-b551ca1e8c6e'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:57.992129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-ba448e07-f84c-4285-8748-b551ca1e8c6e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c0d1e5a9ecb84d10883a49e351a5ad4f', 'request_id': 'req-ba448e07-f84c-4285-8748-b551ca1e8c6e'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:57.998366 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-ba448e07-f84c-4285-8748-b551ca1e8c6e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c0d1e5a9ecb84d10883a49e351a5ad4f', 'request_id': 'req-ba448e07-f84c-4285-8748-b551ca1e8c6e'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:58.012138 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-ba448e07-f84c-4285-8748-b551ca1e8c6e admin admin] The token cache is being invalidate because project c0d1e5a9ecb84d10883a49e351a5ad4f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:26:58.012609 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-ba448e07-f84c-4285-8748-b551ca1e8c6e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:58.013351 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-ba448e07-f84c-4285-8748-b551ca1e8c6e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:58.014946 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1287/2585] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:26:57 2022] DELETE /identity/v3/projects/c0d1e5a9ecb84d10883a49e351a5ad4f => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:58.020067 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b65c7c4-7374-41a5-ae9c-f1a6bc9bfb33 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:58.020230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b65c7c4-7374-41a5-ae9c-f1a6bc9bfb33 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:58.020506 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b65c7c4-7374-41a5-ae9c-f1a6bc9bfb33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:58.040242 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-5b65c7c4-7374-41a5-ae9c-f1a6bc9bfb33 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:58.041516 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-5b65c7c4-7374-41a5-ae9c-f1a6bc9bfb33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:58.051797 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:58.103560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1299/2586] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:58.113355 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8479f9-be63-4ac5-96aa-4120ab2cfa1d None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:58.113647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8479f9-be63-4ac5-96aa-4120ab2cfa1d None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:58.113964 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8479f9-be63-4ac5-96aa-4120ab2cfa1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:58.122295 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:58.136289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-eb8479f9-be63-4ac5-96aa-4120ab2cfa1d None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:58.137971 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-eb8479f9-be63-4ac5-96aa-4120ab2cfa1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:58.146259 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1288/2587] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:58.150689 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b65c7c4-7374-41a5-ae9c-f1a6bc9bfb33 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:58.161221 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b65c7c4-7374-41a5-ae9c-f1a6bc9bfb33 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:58.165938 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07424398-3ade-4368-b936-ebd6b57c1dbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:58.166613 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07424398-3ade-4368-b936-ebd6b57c1dbc admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:58.167059 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07424398-3ade-4368-b936-ebd6b57c1dbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:58.167240 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07424398-3ade-4368-b936-ebd6b57c1dbc admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:58.167896 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07424398-3ade-4368-b936-ebd6b57c1dbc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:26:58.168196 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07424398-3ade-4368-b936-ebd6b57c1dbc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:58.169810 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07424398-3ade-4368-b936-ebd6b57c1dbc admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:58.182497 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1300/2588] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:26:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:26:58.187307 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8479f9-be63-4ac5-96aa-4120ab2cfa1d None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:58.197496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8479f9-be63-4ac5-96aa-4120ab2cfa1d None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:58.202162 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ca6fab-aab8-463c-b399-f21db58bc1ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:58.202753 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ca6fab-aab8-463c-b399-f21db58bc1ca admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:58.203029 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ca6fab-aab8-463c-b399-f21db58bc1ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:58.203297 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ca6fab-aab8-463c-b399-f21db58bc1ca admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:58.203953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ca6fab-aab8-463c-b399-f21db58bc1ca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:58.205942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ca6fab-aab8-463c-b399-f21db58bc1ca admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:58.239001 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1289/2589] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:26:58 2022] POST /identity/v3/projects => generated 373 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:26:58.243791 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07424398-3ade-4368-b936-ebd6b57c1dbc admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:58.254343 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07424398-3ade-4368-b936-ebd6b57c1dbc admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:58.311548 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0076eadf-b579-4577-ad93-0b55a5ca5208 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:58.312146 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0076eadf-b579-4577-ad93-0b55a5ca5208 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:58.312439 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0076eadf-b579-4577-ad93-0b55a5ca5208 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:58.312708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0076eadf-b579-4577-ad93-0b55a5ca5208 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:58.314001 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0076eadf-b579-4577-ad93-0b55a5ca5208 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:58.317764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0076eadf-b579-4577-ad93-0b55a5ca5208 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:58.339281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1301/2590] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:26:58 2022] POST /identity/v3/users => generated 287 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:26:58.340536 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ca6fab-aab8-463c-b399-f21db58bc1ca admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:58.351504 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ca6fab-aab8-463c-b399-f21db58bc1ca admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:58.356576 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c9753e-1786-45ea-9c72-ebc50fcd26f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:58.357181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c9753e-1786-45ea-9c72-ebc50fcd26f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:58.357465 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c9753e-1786-45ea-9c72-ebc50fcd26f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:58.357765 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c9753e-1786-45ea-9c72-ebc50fcd26f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:58.358424 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c9753e-1786-45ea-9c72-ebc50fcd26f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:58.360070 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c9753e-1786-45ea-9c72-ebc50fcd26f3 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:58.371231 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1290/2591] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:58 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:58.375015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0076eadf-b579-4577-ad93-0b55a5ca5208 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:58.385398 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0076eadf-b579-4577-ad93-0b55a5ca5208 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:58.389915 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f61b39-1a56-4e4a-904a-2bbd4fa7de82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:58.390516 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f61b39-1a56-4e4a-904a-2bbd4fa7de82 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:58.390795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f61b39-1a56-4e4a-904a-2bbd4fa7de82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:58.391067 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f61b39-1a56-4e4a-904a-2bbd4fa7de82 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:58.391732 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f61b39-1a56-4e4a-904a-2bbd4fa7de82 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:58.393435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f61b39-1a56-4e4a-904a-2bbd4fa7de82 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:58.404516 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1302/2592] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:26:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:26:58.414146 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c9753e-1786-45ea-9c72-ebc50fcd26f3 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:58.426362 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c9753e-1786-45ea-9c72-ebc50fcd26f3 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:58.431892 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1694503-9978-41b8-9146-d2ce90419670 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:58.432729 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1694503-9978-41b8-9146-d2ce90419670 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:58.433079 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1694503-9978-41b8-9146-d2ce90419670 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:58.433423 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1694503-9978-41b8-9146-d2ce90419670 admin admin] PATH_INFO: `/v3/projects/d70fde5da741495e8eb5f199cd42d7a1/users/26aab925dbc7428e9d5f79b0866a6992/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:58.449350 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1694503-9978-41b8-9146-d2ce90419670 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d70fde5da741495e8eb5f199cd42d7a1, user_id=26aab925dbc7428e9d5f79b0866a6992, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:58.452098 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1694503-9978-41b8-9146-d2ce90419670 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:58.465663 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1291/2593] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:26:58 2022] PUT /identity/v3/projects/d70fde5da741495e8eb5f199cd42d7a1/users/26aab925dbc7428e9d5f79b0866a6992/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:58.476670 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e221fa08-f9ed-46d4-9bb0-f911e566e5f0 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:58.477531 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e221fa08-f9ed-46d4-9bb0-f911e566e5f0 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:58.477839 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e221fa08-f9ed-46d4-9bb0-f911e566e5f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:58.505953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-e221fa08-f9ed-46d4-9bb0-f911e566e5f0 None None] MFA Rules not processed for user `26aab925dbc7428e9d5f79b0866a6992`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:58.507152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-e221fa08-f9ed-46d4-9bb0-f911e566e5f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:58.574507 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1303/2594] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:58 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:26:58.588487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1694503-9978-41b8-9146-d2ce90419670 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:58.590315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-f1694503-9978-41b8-9146-d2ce90419670 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:58.648218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1694503-9978-41b8-9146-d2ce90419670 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:58.651707 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99632bb5-ac6d-4c40-bde2-6ea3816e16fd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:58.652310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99632bb5-ac6d-4c40-bde2-6ea3816e16fd service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:58.652588 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99632bb5-ac6d-4c40-bde2-6ea3816e16fd service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:58.652871 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99632bb5-ac6d-4c40-bde2-6ea3816e16fd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:58.662174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99632bb5-ac6d-4c40-bde2-6ea3816e16fd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:58.663606 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99632bb5-ac6d-4c40-bde2-6ea3816e16fd service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:58.672637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1292/2595] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:26:58 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:26:58.699027 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e221fa08-f9ed-46d4-9bb0-f911e566e5f0 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:58.712247 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:58.763732 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e221fa08-f9ed-46d4-9bb0-f911e566e5f0 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:58.768881 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d6a727f-401a-4ddc-81c3-4d45e5851529 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:58.769500 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6a727f-401a-4ddc-81c3-4d45e5851529 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:58.769800 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6a727f-401a-4ddc-81c3-4d45e5851529 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:58.770074 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6a727f-401a-4ddc-81c3-4d45e5851529 admin admin] PATH_INFO: `/v3/users/26aab925dbc7428e9d5f79b0866a6992` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:58.771191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d6a727f-401a-4ddc-81c3-4d45e5851529 admin admin] RBAC: Authorizing `identity:delete_user(user_id=26aab925dbc7428e9d5f79b0866a6992)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:58.773072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d6a727f-401a-4ddc-81c3-4d45e5851529 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:58.816939 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8d6a727f-401a-4ddc-81c3-4d45e5851529 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '26aab925dbc7428e9d5f79b0866a6992'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:58.823585 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8d6a727f-401a-4ddc-81c3-4d45e5851529 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '26aab925dbc7428e9d5f79b0866a6992'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:58.824243 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8d6a727f-401a-4ddc-81c3-4d45e5851529 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '26aab925dbc7428e9d5f79b0866a6992'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:58.831551 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8d6a727f-401a-4ddc-81c3-4d45e5851529 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '26aab925dbc7428e9d5f79b0866a6992'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:58.843314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8d6a727f-401a-4ddc-81c3-4d45e5851529 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '26aab925dbc7428e9d5f79b0866a6992'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:58.849107 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1304/2596] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:26:58 2022] DELETE /identity/v3/users/26aab925dbc7428e9d5f79b0866a6992 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:26:58.853599 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3937e1-9a95-4dc6-a818-43b39cbe51e9 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:58.853923 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3937e1-9a95-4dc6-a818-43b39cbe51e9 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:58.854198 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3937e1-9a95-4dc6-a818-43b39cbe51e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:58.873349 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-8e3937e1-9a95-4dc6-a818-43b39cbe51e9 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:26:58.874583 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-8e3937e1-9a95-4dc6-a818-43b39cbe51e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:58.931306 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1293/2597] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:26:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:26:58.941529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d6a727f-401a-4ddc-81c3-4d45e5851529 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:58.942918 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-8d6a727f-401a-4ddc-81c3-4d45e5851529 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:59.006956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d6a727f-401a-4ddc-81c3-4d45e5851529 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:59.010602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc49491-b425-4e90-a3d3-d9377de3c608 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:59.011205 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc49491-b425-4e90-a3d3-d9377de3c608 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:59.011481 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc49491-b425-4e90-a3d3-d9377de3c608 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:59.011751 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc49491-b425-4e90-a3d3-d9377de3c608 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:59.020956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc49491-b425-4e90-a3d3-d9377de3c608 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:59.022418 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc49491-b425-4e90-a3d3-d9377de3c608 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:59.031658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1305/2598] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:26:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:26:59.306541 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3937e1-9a95-4dc6-a818-43b39cbe51e9 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:26:59.308030 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-8e3937e1-9a95-4dc6-a818-43b39cbe51e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:26:59.321985 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3937e1-9a95-4dc6-a818-43b39cbe51e9 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:26:59.326746 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4bcca9-4b02-465f-b5df-11da5214edae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:26:59.327482 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4bcca9-4b02-465f-b5df-11da5214edae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:26:59.327859 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4bcca9-4b02-465f-b5df-11da5214edae admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:26:59.328568 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4bcca9-4b02-465f-b5df-11da5214edae admin admin] PATH_INFO: `/v3/projects/d70fde5da741495e8eb5f199cd42d7a1` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:26:59.329452 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4bcca9-4b02-465f-b5df-11da5214edae admin admin] RBAC: Authorizing `identity:delete_project(project_id=d70fde5da741495e8eb5f199cd42d7a1)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:26:59.331998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4bcca9-4b02-465f-b5df-11da5214edae admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:26:59.371487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:26:59.381871 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-cd4bcca9-4b02-465f-b5df-11da5214edae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd70fde5da741495e8eb5f199cd42d7a1', 'request_id': 'req-cd4bcca9-4b02-465f-b5df-11da5214edae'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:59.387356 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-cd4bcca9-4b02-465f-b5df-11da5214edae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd70fde5da741495e8eb5f199cd42d7a1', 'request_id': 'req-cd4bcca9-4b02-465f-b5df-11da5214edae'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:59.394188 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-cd4bcca9-4b02-465f-b5df-11da5214edae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd70fde5da741495e8eb5f199cd42d7a1', 'request_id': 'req-cd4bcca9-4b02-465f-b5df-11da5214edae'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:59.406447 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-cd4bcca9-4b02-465f-b5df-11da5214edae admin admin] The token cache is being invalidate because project d70fde5da741495e8eb5f199cd42d7a1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:26:59.406763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-cd4bcca9-4b02-465f-b5df-11da5214edae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:59.407404 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-cd4bcca9-4b02-465f-b5df-11da5214edae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:26:59.409207 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1294/2599] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:26:59 2022] DELETE /identity/v3/projects/d70fde5da741495e8eb5f199cd42d7a1 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:27:06.641820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc49491-b425-4e90-a3d3-d9377de3c608 service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:06.650779 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc49491-b425-4e90-a3d3-d9377de3c608 service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:06.683043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:27:06.705409 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a0bdbc-c3de-4c36-ad3d-8d6de1b3264f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:06.706063 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a0bdbc-c3de-4c36-ad3d-8d6de1b3264f service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:06.706344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a0bdbc-c3de-4c36-ad3d-8d6de1b3264f service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:06.706623 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a0bdbc-c3de-4c36-ad3d-8d6de1b3264f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:06.708330 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c7a0bdbc-c3de-4c36-ad3d-8d6de1b3264f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:06.780009 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a0bdbc-c3de-4c36-ad3d-8d6de1b3264f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:06.781443 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a0bdbc-c3de-4c36-ad3d-8d6de1b3264f service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:06.793774 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1306/2600] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:27:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:27:09.403520 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=11,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:27:09.937234 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4bcca9-4b02-465f-b5df-11da5214edae admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:09.938769 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-cd4bcca9-4b02-465f-b5df-11da5214edae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:09.954433 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4bcca9-4b02-465f-b5df-11da5214edae admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:09.959966 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8113fc5a-8e3d-4043-b7ef-24d56a078ec9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:09.960746 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8113fc5a-8e3d-4043-b7ef-24d56a078ec9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:09.961024 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8113fc5a-8e3d-4043-b7ef-24d56a078ec9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:09.961297 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8113fc5a-8e3d-4043-b7ef-24d56a078ec9 admin admin] PATH_INFO: `/v3/users/7c56e7333ce4450b8cbd5caeafc73947` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:09.962441 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8113fc5a-8e3d-4043-b7ef-24d56a078ec9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7c56e7333ce4450b8cbd5caeafc73947)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:09.964335 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8113fc5a-8e3d-4043-b7ef-24d56a078ec9 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:10.016245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8113fc5a-8e3d-4043-b7ef-24d56a078ec9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7c56e7333ce4450b8cbd5caeafc73947'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:10.023822 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8113fc5a-8e3d-4043-b7ef-24d56a078ec9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7c56e7333ce4450b8cbd5caeafc73947'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:10.024486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8113fc5a-8e3d-4043-b7ef-24d56a078ec9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7c56e7333ce4450b8cbd5caeafc73947'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:10.032469 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8113fc5a-8e3d-4043-b7ef-24d56a078ec9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7c56e7333ce4450b8cbd5caeafc73947'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:10.049603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8113fc5a-8e3d-4043-b7ef-24d56a078ec9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7c56e7333ce4450b8cbd5caeafc73947'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:10.056241 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1295/2601] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:27:09 2022] DELETE /identity/v3/users/7c56e7333ce4450b8cbd5caeafc73947 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:27:10.232329 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a0bdbc-c3de-4c36-ad3d-8d6de1b3264f service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:10.244296 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a0bdbc-c3de-4c36-ad3d-8d6de1b3264f service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:10.309930 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e5a8e6-1ecd-4961-a538-19439921ed6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:10.310569 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e5a8e6-1ecd-4961-a538-19439921ed6b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:10.310850 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e5a8e6-1ecd-4961-a538-19439921ed6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:10.311124 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e5a8e6-1ecd-4961-a538-19439921ed6b admin admin] PATH_INFO: `/v3/projects/312fa89a4d9b43c5919b7bad5411002a` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:10.312237 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e5a8e6-1ecd-4961-a538-19439921ed6b admin admin] RBAC: Authorizing `identity:delete_project(project_id=312fa89a4d9b43c5919b7bad5411002a)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:10.314185 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e5a8e6-1ecd-4961-a538-19439921ed6b admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:10.359874 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-f0e5a8e6-1ecd-4961-a538-19439921ed6b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '312fa89a4d9b43c5919b7bad5411002a', 'request_id': 'req-f0e5a8e6-1ecd-4961-a538-19439921ed6b'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:10.365916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-f0e5a8e6-1ecd-4961-a538-19439921ed6b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '312fa89a4d9b43c5919b7bad5411002a', 'request_id': 'req-f0e5a8e6-1ecd-4961-a538-19439921ed6b'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:10.372186 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-f0e5a8e6-1ecd-4961-a538-19439921ed6b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '312fa89a4d9b43c5919b7bad5411002a', 'request_id': 'req-f0e5a8e6-1ecd-4961-a538-19439921ed6b'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:10.383700 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-f0e5a8e6-1ecd-4961-a538-19439921ed6b admin admin] The token cache is being invalidate because project 312fa89a4d9b43c5919b7bad5411002a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:27:10.383987 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-f0e5a8e6-1ecd-4961-a538-19439921ed6b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:10.384640 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-f0e5a8e6-1ecd-4961-a538-19439921ed6b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:10.386144 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1307/2602] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:27:10 2022] DELETE /identity/v3/projects/312fa89a4d9b43c5919b7bad5411002a => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:27:10.391291 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abecac04-4193-4c21-b3a8-434628f379f1 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:10.391579 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abecac04-4193-4c21-b3a8-434628f379f1 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:10.391853 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abecac04-4193-4c21-b3a8-434628f379f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:10.412871 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-abecac04-4193-4c21-b3a8-434628f379f1 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:27:10.414122 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-abecac04-4193-4c21-b3a8-434628f379f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:10.472933 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1296/2603] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:27:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:27:10.480735 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7020404d-26de-4ac3-aa06-6e0245d86516 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:10.481020 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7020404d-26de-4ac3-aa06-6e0245d86516 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:10.481295 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7020404d-26de-4ac3-aa06-6e0245d86516 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:10.501304 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-7020404d-26de-4ac3-aa06-6e0245d86516 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:27:10.502580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-7020404d-26de-4ac3-aa06-6e0245d86516 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:10.511243 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1308/2604] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:27:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:27:10.515412 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abecac04-4193-4c21-b3a8-434628f379f1 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:10.525139 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abecac04-4193-4c21-b3a8-434628f379f1 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:10.529585 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6881384b-7fb3-4482-9edb-57bdc41b7e11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:10.530198 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6881384b-7fb3-4482-9edb-57bdc41b7e11 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:10.530473 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6881384b-7fb3-4482-9edb-57bdc41b7e11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:10.530742 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6881384b-7fb3-4482-9edb-57bdc41b7e11 admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:10.531382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6881384b-7fb3-4482-9edb-57bdc41b7e11 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:27:10.531679 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6881384b-7fb3-4482-9edb-57bdc41b7e11 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:10.533975 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6881384b-7fb3-4482-9edb-57bdc41b7e11 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:10.545772 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1297/2605] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:27:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:27:10.550490 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7020404d-26de-4ac3-aa06-6e0245d86516 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:10.559435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7020404d-26de-4ac3-aa06-6e0245d86516 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:10.563974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16015f86-ea61-4929-8a55-1fe620bd5b04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:10.564555 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16015f86-ea61-4929-8a55-1fe620bd5b04 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:10.564828 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16015f86-ea61-4929-8a55-1fe620bd5b04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:10.565097 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16015f86-ea61-4929-8a55-1fe620bd5b04 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:10.565716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16015f86-ea61-4929-8a55-1fe620bd5b04 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:10.567611 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16015f86-ea61-4929-8a55-1fe620bd5b04 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:10.571944 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:27:10.594988 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1309/2606] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:27:10 2022] POST /identity/v3/projects => generated 383 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:27:10.599591 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6881384b-7fb3-4482-9edb-57bdc41b7e11 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:10.609376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6881384b-7fb3-4482-9edb-57bdc41b7e11 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:10.670792 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5aa0e5a-1a29-481a-ac00-88e951f92663 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:10.671311 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5aa0e5a-1a29-481a-ac00-88e951f92663 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:10.671595 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5aa0e5a-1a29-481a-ac00-88e951f92663 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:10.671869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5aa0e5a-1a29-481a-ac00-88e951f92663 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:10.672539 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5aa0e5a-1a29-481a-ac00-88e951f92663 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:10.674506 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5aa0e5a-1a29-481a-ac00-88e951f92663 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:10.682607 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:27:10.697593 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1298/2607] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:27:10 2022] POST /identity/v3/users => generated 292 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:27:10.701613 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16015f86-ea61-4929-8a55-1fe620bd5b04 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:10.712094 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16015f86-ea61-4929-8a55-1fe620bd5b04 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:10.716602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2794e90-7d7c-415e-bf4c-898003fd3dcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:10.717183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2794e90-7d7c-415e-bf4c-898003fd3dcc admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:10.717453 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2794e90-7d7c-415e-bf4c-898003fd3dcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:10.717723 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2794e90-7d7c-415e-bf4c-898003fd3dcc admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:10.718388 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2794e90-7d7c-415e-bf4c-898003fd3dcc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:10.719945 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2794e90-7d7c-415e-bf4c-898003fd3dcc admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:10.730083 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1310/2608] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:27:10 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:27:10.733780 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5aa0e5a-1a29-481a-ac00-88e951f92663 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:10.743662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5aa0e5a-1a29-481a-ac00-88e951f92663 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:10.748259 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbddae86-f3b1-4f3e-8012-066e056aecad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:10.748851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbddae86-f3b1-4f3e-8012-066e056aecad admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:10.749152 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbddae86-f3b1-4f3e-8012-066e056aecad admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:10.749413 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbddae86-f3b1-4f3e-8012-066e056aecad admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:10.750097 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbddae86-f3b1-4f3e-8012-066e056aecad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:10.751705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbddae86-f3b1-4f3e-8012-066e056aecad admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:10.762186 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1299/2609] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:27:10 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:27:10.766123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2794e90-7d7c-415e-bf4c-898003fd3dcc admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:10.776190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2794e90-7d7c-415e-bf4c-898003fd3dcc admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:10.780754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c65b86-c389-4499-892b-ad76c04e8e3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:10.781381 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c65b86-c389-4499-892b-ad76c04e8e3a admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:10.781653 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c65b86-c389-4499-892b-ad76c04e8e3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:10.781953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c65b86-c389-4499-892b-ad76c04e8e3a admin admin] PATH_INFO: `/v3/projects/d7c74ffec54740eca3cd6b707f1cb732/users/dfb35bfba72546f3853b9dd9ef7f5078/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:10.796979 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c65b86-c389-4499-892b-ad76c04e8e3a admin admin] RBAC: Authorizing `identity:create_grant(project_id=d7c74ffec54740eca3cd6b707f1cb732, user_id=dfb35bfba72546f3853b9dd9ef7f5078, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:10.800261 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c65b86-c389-4499-892b-ad76c04e8e3a admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:10.812880 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1311/2610] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:27:10 2022] PUT /identity/v3/projects/d7c74ffec54740eca3cd6b707f1cb732/users/dfb35bfba72546f3853b9dd9ef7f5078/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:27:10.818520 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ca52dc-1b26-48e0-bcda-ed8ed7f38263 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:10.818802 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ca52dc-1b26-48e0-bcda-ed8ed7f38263 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:10.819074 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ca52dc-1b26-48e0-bcda-ed8ed7f38263 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:10.837192 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-93ca52dc-1b26-48e0-bcda-ed8ed7f38263 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:27:10.838418 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-93ca52dc-1b26-48e0-bcda-ed8ed7f38263 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:10.894797 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1300/2611] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:27:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:27:10.905385 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c65b86-c389-4499-892b-ad76c04e8e3a admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:10.907116 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d9c65b86-c389-4499-892b-ad76c04e8e3a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:10.963256 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c65b86-c389-4499-892b-ad76c04e8e3a admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:10.966895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb58d25-c94a-4431-acfb-d3e0284db04a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:10.967501 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb58d25-c94a-4431-acfb-d3e0284db04a service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:10.967806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb58d25-c94a-4431-acfb-d3e0284db04a service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:10.968080 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb58d25-c94a-4431-acfb-d3e0284db04a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:10.976960 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb58d25-c94a-4431-acfb-d3e0284db04a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:10.978431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb58d25-c94a-4431-acfb-d3e0284db04a service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:10.988288 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1312/2612] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:27:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:27:14.935309 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc21ef56-b9c1-493d-8423-ee9d3ef6eaf9 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:14.935785 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc21ef56-b9c1-493d-8423-ee9d3ef6eaf9 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:14.935891 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc21ef56-b9c1-493d-8423-ee9d3ef6eaf9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:14.967448 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-cc21ef56-b9c1-493d-8423-ee9d3ef6eaf9 None None] MFA Rules not processed for user `dfb35bfba72546f3853b9dd9ef7f5078`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:27:14.968648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-cc21ef56-b9c1-493d-8423-ee9d3ef6eaf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:15.032016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1301/2613] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:27:14 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:27:15.047217 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb58d25-c94a-4431-acfb-d3e0284db04a service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:15.048630 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-abb58d25-c94a-4431-acfb-d3e0284db04a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:15.121973 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb58d25-c94a-4431-acfb-d3e0284db04a service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:15.127744 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2cb726-b39f-4777-8db8-74c48680d856 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:15.128344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2cb726-b39f-4777-8db8-74c48680d856 service nova] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:15.128620 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2cb726-b39f-4777-8db8-74c48680d856 service nova] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:15.129055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2cb726-b39f-4777-8db8-74c48680d856 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:15.137125 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2cb726-b39f-4777-8db8-74c48680d856 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:15.138726 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2cb726-b39f-4777-8db8-74c48680d856 service nova] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:15.148112 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1313/2614] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:27:15 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:27:16.570009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98be91fc-886b-4fad-a0cd-86bf31f9bdea None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:16.570499 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98be91fc-886b-4fad-a0cd-86bf31f9bdea None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:16.570597 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98be91fc-886b-4fad-a0cd-86bf31f9bdea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:16.589457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-98be91fc-886b-4fad-a0cd-86bf31f9bdea None None] MFA Rules not processed for user `dfb35bfba72546f3853b9dd9ef7f5078`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:27:16.590685 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-98be91fc-886b-4fad-a0cd-86bf31f9bdea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:16.597975 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1302/2615] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:27:16 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:27:16.912080 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2cb726-b39f-4777-8db8-74c48680d856 service nova] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:16.913511 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-8a2cb726-b39f-4777-8db8-74c48680d856 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:16.925892 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2cb726-b39f-4777-8db8-74c48680d856 service nova] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:16.929416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d46f9a-ca07-4c5e-97c4-07b6a71a4065 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:16.930189 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d46f9a-ca07-4c5e-97c4-07b6a71a4065 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:16.930354 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d46f9a-ca07-4c5e-97c4-07b6a71a4065 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:16.930627 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d46f9a-ca07-4c5e-97c4-07b6a71a4065 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:16.932436 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-29d46f9a-ca07-4c5e-97c4-07b6a71a4065 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:16.942683 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d46f9a-ca07-4c5e-97c4-07b6a71a4065 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:16.944104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d46f9a-ca07-4c5e-97c4-07b6a71a4065 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:16.952566 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1314/2616] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:27:16 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:27:26.583972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:INSERT=3,keystone:SELECT=69 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:27:26.952862 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=87,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:27:30.976778 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98be91fc-886b-4fad-a0cd-86bf31f9bdea None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:30.978248 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-98be91fc-886b-4fad-a0cd-86bf31f9bdea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:30.992019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98be91fc-886b-4fad-a0cd-86bf31f9bdea None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:30.996470 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701c2736-4b40-4708-822f-7de157c9dab3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:30.997091 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701c2736-4b40-4708-822f-7de157c9dab3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:30.997370 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701c2736-4b40-4708-822f-7de157c9dab3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:30.997642 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701c2736-4b40-4708-822f-7de157c9dab3 admin admin] PATH_INFO: `/v3/users/0c510124846b47598c04441bf5dd151d` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:30.998830 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-701c2736-4b40-4708-822f-7de157c9dab3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0c510124846b47598c04441bf5dd151d)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:31.000862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-701c2736-4b40-4708-822f-7de157c9dab3 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:31.047354 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-701c2736-4b40-4708-822f-7de157c9dab3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0c510124846b47598c04441bf5dd151d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:31.054682 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-701c2736-4b40-4708-822f-7de157c9dab3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0c510124846b47598c04441bf5dd151d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:31.055358 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-701c2736-4b40-4708-822f-7de157c9dab3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0c510124846b47598c04441bf5dd151d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:31.063070 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-701c2736-4b40-4708-822f-7de157c9dab3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0c510124846b47598c04441bf5dd151d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:31.075764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-701c2736-4b40-4708-822f-7de157c9dab3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0c510124846b47598c04441bf5dd151d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:31.081647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1303/2617] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:27:30 2022] DELETE /identity/v3/users/0c510124846b47598c04441bf5dd151d => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:27:31.086271 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c102c13b-94e5-4766-846e-d4e5c060c1ad None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:31.086725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c102c13b-94e5-4766-846e-d4e5c060c1ad None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:31.086843 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c102c13b-94e5-4766-846e-d4e5c060c1ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:31.106150 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-c102c13b-94e5-4766-846e-d4e5c060c1ad None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:27:31.107426 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c102c13b-94e5-4766-846e-d4e5c060c1ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:31.168332 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1315/2618] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:27:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:27:31.178874 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701c2736-4b40-4708-822f-7de157c9dab3 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:31.180339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-701c2736-4b40-4708-822f-7de157c9dab3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:31.239717 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701c2736-4b40-4708-822f-7de157c9dab3 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:31.243437 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cd7feb-a871-4369-a5e3-7f029db558e2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:31.244047 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cd7feb-a871-4369-a5e3-7f029db558e2 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:31.244328 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cd7feb-a871-4369-a5e3-7f029db558e2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:31.244604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cd7feb-a871-4369-a5e3-7f029db558e2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:31.254538 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cd7feb-a871-4369-a5e3-7f029db558e2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:31.256067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cd7feb-a871-4369-a5e3-7f029db558e2 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:31.266513 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1304/2619] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:27:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:27:31.537320 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c102c13b-94e5-4766-846e-d4e5c060c1ad None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:31.538814 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c102c13b-94e5-4766-846e-d4e5c060c1ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:31.551762 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c102c13b-94e5-4766-846e-d4e5c060c1ad None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:31.556299 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d00fdec-de2c-4a44-b531-e9975fa2cca7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:31.556939 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d00fdec-de2c-4a44-b531-e9975fa2cca7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:31.557222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d00fdec-de2c-4a44-b531-e9975fa2cca7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:31.557501 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d00fdec-de2c-4a44-b531-e9975fa2cca7 admin admin] PATH_INFO: `/v3/projects/ae2f2bea79ab48c5bd28339fdac86fb3` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:31.558658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d00fdec-de2c-4a44-b531-e9975fa2cca7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ae2f2bea79ab48c5bd28339fdac86fb3)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:31.560603 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d00fdec-de2c-4a44-b531-e9975fa2cca7 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:31.612995 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6d00fdec-de2c-4a44-b531-e9975fa2cca7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ae2f2bea79ab48c5bd28339fdac86fb3', 'request_id': 'req-6d00fdec-de2c-4a44-b531-e9975fa2cca7'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:31.622972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6d00fdec-de2c-4a44-b531-e9975fa2cca7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ae2f2bea79ab48c5bd28339fdac86fb3', 'request_id': 'req-6d00fdec-de2c-4a44-b531-e9975fa2cca7'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:31.630099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6d00fdec-de2c-4a44-b531-e9975fa2cca7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ae2f2bea79ab48c5bd28339fdac86fb3', 'request_id': 'req-6d00fdec-de2c-4a44-b531-e9975fa2cca7'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:31.652299 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6d00fdec-de2c-4a44-b531-e9975fa2cca7 admin admin] The token cache is being invalidate because project ae2f2bea79ab48c5bd28339fdac86fb3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:27:31.652592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6d00fdec-de2c-4a44-b531-e9975fa2cca7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:31.653241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6d00fdec-de2c-4a44-b531-e9975fa2cca7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:31.655188 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1316/2620] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:27:31 2022] DELETE /identity/v3/projects/ae2f2bea79ab48c5bd28339fdac86fb3 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:27:31.659933 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cd7feb-a871-4369-a5e3-7f029db558e2 service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:31.671586 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cd7feb-a871-4369-a5e3-7f029db558e2 service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:31.733523 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046c198d-856e-41ec-b3dd-fdb75204b0c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:31.734200 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046c198d-856e-41ec-b3dd-fdb75204b0c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:31.734480 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046c198d-856e-41ec-b3dd-fdb75204b0c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:31.734755 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046c198d-856e-41ec-b3dd-fdb75204b0c1 admin admin] PATH_INFO: `/v3/users/ffd308127d1946debcb689908632c7b0` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:31.735885 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046c198d-856e-41ec-b3dd-fdb75204b0c1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ffd308127d1946debcb689908632c7b0)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:31.738123 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046c198d-856e-41ec-b3dd-fdb75204b0c1 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:31.755643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:27:31.797191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-046c198d-856e-41ec-b3dd-fdb75204b0c1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ffd308127d1946debcb689908632c7b0'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:31.804870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-046c198d-856e-41ec-b3dd-fdb75204b0c1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ffd308127d1946debcb689908632c7b0'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:31.805670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-046c198d-856e-41ec-b3dd-fdb75204b0c1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ffd308127d1946debcb689908632c7b0'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:31.816137 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-046c198d-856e-41ec-b3dd-fdb75204b0c1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ffd308127d1946debcb689908632c7b0'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:31.831137 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-046c198d-856e-41ec-b3dd-fdb75204b0c1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ffd308127d1946debcb689908632c7b0'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:31.837516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1305/2621] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:27:31 2022] DELETE /identity/v3/users/ffd308127d1946debcb689908632c7b0 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:27:32.138188 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d00fdec-de2c-4a44-b531-e9975fa2cca7 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:32.139717 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6d00fdec-de2c-4a44-b531-e9975fa2cca7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:32.159810 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d00fdec-de2c-4a44-b531-e9975fa2cca7 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:32.228333 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a9d414-ab5c-4c4c-b105-5bd72dcf4835 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:32.229481 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a9d414-ab5c-4c4c-b105-5bd72dcf4835 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:32.229938 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a9d414-ab5c-4c4c-b105-5bd72dcf4835 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:32.230389 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a9d414-ab5c-4c4c-b105-5bd72dcf4835 admin admin] PATH_INFO: `/v3/projects/7614a1b21d144efeb90a6e7ced0b9ed0` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:32.231749 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a9d414-ab5c-4c4c-b105-5bd72dcf4835 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7614a1b21d144efeb90a6e7ced0b9ed0)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:32.233892 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a9d414-ab5c-4c4c-b105-5bd72dcf4835 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:32.271043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:27:32.289828 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-56a9d414-ab5c-4c4c-b105-5bd72dcf4835 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7614a1b21d144efeb90a6e7ced0b9ed0', 'request_id': 'req-56a9d414-ab5c-4c4c-b105-5bd72dcf4835'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:32.296219 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-56a9d414-ab5c-4c4c-b105-5bd72dcf4835 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7614a1b21d144efeb90a6e7ced0b9ed0', 'request_id': 'req-56a9d414-ab5c-4c4c-b105-5bd72dcf4835'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:32.302934 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-56a9d414-ab5c-4c4c-b105-5bd72dcf4835 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7614a1b21d144efeb90a6e7ced0b9ed0', 'request_id': 'req-56a9d414-ab5c-4c4c-b105-5bd72dcf4835'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:32.317187 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-56a9d414-ab5c-4c4c-b105-5bd72dcf4835 admin admin] The token cache is being invalidate because project 7614a1b21d144efeb90a6e7ced0b9ed0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:27:32.317480 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-56a9d414-ab5c-4c4c-b105-5bd72dcf4835 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:32.318149 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-56a9d414-ab5c-4c4c-b105-5bd72dcf4835 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:32.319650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1317/2622] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:27:32 2022] DELETE /identity/v3/projects/7614a1b21d144efeb90a6e7ced0b9ed0 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:27:32.325532 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1e7546-662b-4cff-862e-066bb66bc97e None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:32.326088 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1e7546-662b-4cff-862e-066bb66bc97e None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:32.326185 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1e7546-662b-4cff-862e-066bb66bc97e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:32.347589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-3f1e7546-662b-4cff-862e-066bb66bc97e None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:27:32.348799 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-3f1e7546-662b-4cff-862e-066bb66bc97e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:32.414443 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1306/2623] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:27:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:27:32.422565 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5b4009-1ff3-4448-8b25-fa52d64540dd None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:32.423781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5b4009-1ff3-4448-8b25-fa52d64540dd None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:32.423781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5b4009-1ff3-4448-8b25-fa52d64540dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:32.443830 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-3f5b4009-1ff3-4448-8b25-fa52d64540dd None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:27:32.445029 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-3f5b4009-1ff3-4448-8b25-fa52d64540dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:32.453296 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1318/2624] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:27:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:27:32.457457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f1e7546-662b-4cff-862e-066bb66bc97e None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:32.467289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f1e7546-662b-4cff-862e-066bb66bc97e None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:32.472143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d733f0-572d-4b99-94e8-520cc9dfff92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:32.472753 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d733f0-572d-4b99-94e8-520cc9dfff92 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:32.473062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d733f0-572d-4b99-94e8-520cc9dfff92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:32.473335 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d733f0-572d-4b99-94e8-520cc9dfff92 admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:32.474023 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d733f0-572d-4b99-94e8-520cc9dfff92 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:27:32.474327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d733f0-572d-4b99-94e8-520cc9dfff92 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:32.476023 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d733f0-572d-4b99-94e8-520cc9dfff92 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:32.491992 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1307/2625] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:27:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:27:32.493085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5b4009-1ff3-4448-8b25-fa52d64540dd None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:32.502751 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5b4009-1ff3-4448-8b25-fa52d64540dd None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:32.507371 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9448c2e-1f6c-4a61-a255-b9b4f97d1c12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:32.507959 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9448c2e-1f6c-4a61-a255-b9b4f97d1c12 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:32.508233 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9448c2e-1f6c-4a61-a255-b9b4f97d1c12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:32.508504 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9448c2e-1f6c-4a61-a255-b9b4f97d1c12 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:32.509252 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9448c2e-1f6c-4a61-a255-b9b4f97d1c12 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:32.511147 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9448c2e-1f6c-4a61-a255-b9b4f97d1c12 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:32.542290 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1319/2626] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:27:32 2022] POST /identity/v3/projects => generated 407 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:27:32.543342 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d733f0-572d-4b99-94e8-520cc9dfff92 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:32.554082 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d733f0-572d-4b99-94e8-520cc9dfff92 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:32.623205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b34688-a404-43f7-ab94-9636a344ab53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:32.624643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b34688-a404-43f7-ab94-9636a344ab53 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:32.624643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b34688-a404-43f7-ab94-9636a344ab53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:32.624643 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b34688-a404-43f7-ab94-9636a344ab53 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:32.625196 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31b34688-a404-43f7-ab94-9636a344ab53 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:32.627037 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31b34688-a404-43f7-ab94-9636a344ab53 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:32.651758 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1308/2627] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:27:32 2022] POST /identity/v3/users => generated 304 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:27:32.653614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9448c2e-1f6c-4a61-a255-b9b4f97d1c12 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:32.663837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9448c2e-1f6c-4a61-a255-b9b4f97d1c12 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:32.668631 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a8f4cc-bba6-448e-926a-1e966de4f310 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:32.669217 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a8f4cc-bba6-448e-926a-1e966de4f310 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:32.669489 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a8f4cc-bba6-448e-926a-1e966de4f310 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:32.669792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a8f4cc-bba6-448e-926a-1e966de4f310 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:32.670484 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a8f4cc-bba6-448e-926a-1e966de4f310 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:32.672239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a8f4cc-bba6-448e-926a-1e966de4f310 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:32.683716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1320/2628] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:27:32 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:27:32.687774 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b34688-a404-43f7-ab94-9636a344ab53 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:32.699264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b34688-a404-43f7-ab94-9636a344ab53 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:32.704204 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5576557b-d282-4b85-8ba0-264aac4589fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:32.704842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5576557b-d282-4b85-8ba0-264aac4589fa admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:32.705147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5576557b-d282-4b85-8ba0-264aac4589fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:32.705454 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5576557b-d282-4b85-8ba0-264aac4589fa admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:32.706136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5576557b-d282-4b85-8ba0-264aac4589fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:32.707726 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5576557b-d282-4b85-8ba0-264aac4589fa admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:32.713481 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:DELETE=9,keystone:SELECT=87,keystone:INSERT=4 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:27:32.721937 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1309/2629] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:27:32 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:27:32.724747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a8f4cc-bba6-448e-926a-1e966de4f310 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:32.737552 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a8f4cc-bba6-448e-926a-1e966de4f310 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:32.742597 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed3406a-e3d8-43cc-9a38-b1545e53c6ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:32.743265 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed3406a-e3d8-43cc-9a38-b1545e53c6ee admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:32.743543 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed3406a-e3d8-43cc-9a38-b1545e53c6ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:32.743819 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed3406a-e3d8-43cc-9a38-b1545e53c6ee admin admin] PATH_INFO: `/v3/projects/cb019f622c4b488197b720549a9ffa73/users/9cc9368ddeea4dc997075d1024ec518d/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:32.760284 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed3406a-e3d8-43cc-9a38-b1545e53c6ee admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb019f622c4b488197b720549a9ffa73, user_id=9cc9368ddeea4dc997075d1024ec518d, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:32.762961 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed3406a-e3d8-43cc-9a38-b1545e53c6ee admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:32.777188 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1321/2630] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:27:32 2022] PUT /identity/v3/projects/cb019f622c4b488197b720549a9ffa73/users/9cc9368ddeea4dc997075d1024ec518d/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:27:32.789278 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2500af1c-116e-47fc-8bde-9f0bafbdcec7 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:32.789596 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2500af1c-116e-47fc-8bde-9f0bafbdcec7 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:32.789988 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2500af1c-116e-47fc-8bde-9f0bafbdcec7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:32.819496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-2500af1c-116e-47fc-8bde-9f0bafbdcec7 None None] MFA Rules not processed for user `9cc9368ddeea4dc997075d1024ec518d`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:27:32.820705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-2500af1c-116e-47fc-8bde-9f0bafbdcec7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:32.887078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1310/2631] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:27:32 2022] POST /identity/v3/auth/tokens => generated 3284 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:27:32.891710 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed3406a-e3d8-43cc-9a38-b1545e53c6ee admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:32.960448 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed3406a-e3d8-43cc-9a38-b1545e53c6ee admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:32.965208 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303a71e3-74b5-44ab-9fe6-d1ee042707f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:32.965841 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303a71e3-74b5-44ab-9fe6-d1ee042707f0 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:32.966120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303a71e3-74b5-44ab-9fe6-d1ee042707f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:32.966391 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303a71e3-74b5-44ab-9fe6-d1ee042707f0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:32.967021 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303a71e3-74b5-44ab-9fe6-d1ee042707f0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:32.968895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303a71e3-74b5-44ab-9fe6-d1ee042707f0 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:32.995271 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1322/2632] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:27:32 2022] POST /identity/v3/projects => generated 409 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:27:32.999328 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2500af1c-116e-47fc-8bde-9f0bafbdcec7 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:33.009729 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2500af1c-116e-47fc-8bde-9f0bafbdcec7 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:33.064668 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce244735-93a4-4062-949f-d224a0080e85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:33.065192 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce244735-93a4-4062-949f-d224a0080e85 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:33.065489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce244735-93a4-4062-949f-d224a0080e85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:33.065766 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce244735-93a4-4062-949f-d224a0080e85 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:33.066397 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce244735-93a4-4062-949f-d224a0080e85 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:33.068300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce244735-93a4-4062-949f-d224a0080e85 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:33.088727 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1311/2633] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:27:32 2022] POST /identity/v3/users => generated 311 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:27:33.092744 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303a71e3-74b5-44ab-9fe6-d1ee042707f0 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:33.103840 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303a71e3-74b5-44ab-9fe6-d1ee042707f0 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:33.108416 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f4bb34-4f65-40ee-8e0d-37bce011eb7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:33.108998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f4bb34-4f65-40ee-8e0d-37bce011eb7d admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:33.109270 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f4bb34-4f65-40ee-8e0d-37bce011eb7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:33.109552 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f4bb34-4f65-40ee-8e0d-37bce011eb7d admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:33.110223 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f4bb34-4f65-40ee-8e0d-37bce011eb7d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:33.112412 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f4bb34-4f65-40ee-8e0d-37bce011eb7d admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:33.122795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1323/2634] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:27:33 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:27:33.126816 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce244735-93a4-4062-949f-d224a0080e85 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:33.136572 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce244735-93a4-4062-949f-d224a0080e85 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:33.141096 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d540ed09-cea5-4f77-b15b-ba7465c223ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:33.141763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d540ed09-cea5-4f77-b15b-ba7465c223ab admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:33.142045 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d540ed09-cea5-4f77-b15b-ba7465c223ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:33.142322 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d540ed09-cea5-4f77-b15b-ba7465c223ab admin admin] PATH_INFO: `/v3/projects/cabea523b9aa4ee7a5fb7fadc9c53be8/users/883ea5aef1424b6487a4bb1ba56ad492/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:33.157043 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d540ed09-cea5-4f77-b15b-ba7465c223ab admin admin] RBAC: Authorizing `identity:create_grant(project_id=cabea523b9aa4ee7a5fb7fadc9c53be8, user_id=883ea5aef1424b6487a4bb1ba56ad492, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:33.160240 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d540ed09-cea5-4f77-b15b-ba7465c223ab admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:33.172971 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1312/2635] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:27:33 2022] PUT /identity/v3/projects/cabea523b9aa4ee7a5fb7fadc9c53be8/users/883ea5aef1424b6487a4bb1ba56ad492/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:27:33.176572 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f4bb34-4f65-40ee-8e0d-37bce011eb7d admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:33.186510 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f4bb34-4f65-40ee-8e0d-37bce011eb7d admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:33.206589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:27:33.243457 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540158a8-2eec-48ad-a5d2-0b1e83fe8459 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:33.244037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540158a8-2eec-48ad-a5d2-0b1e83fe8459 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:33.244315 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540158a8-2eec-48ad-a5d2-0b1e83fe8459 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:33.244596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540158a8-2eec-48ad-a5d2-0b1e83fe8459 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:33.245225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540158a8-2eec-48ad-a5d2-0b1e83fe8459 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:33.246827 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540158a8-2eec-48ad-a5d2-0b1e83fe8459 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:33.256391 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1324/2636] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:27:33 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:27:33.260522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d540ed09-cea5-4f77-b15b-ba7465c223ab admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:33.271006 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d540ed09-cea5-4f77-b15b-ba7465c223ab admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:33.275809 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03108bad-9855-409b-8ab1-1caaece1635f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:33.276452 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03108bad-9855-409b-8ab1-1caaece1635f admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:33.276730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03108bad-9855-409b-8ab1-1caaece1635f admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:33.277002 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03108bad-9855-409b-8ab1-1caaece1635f admin admin] PATH_INFO: `/v3/projects/cabea523b9aa4ee7a5fb7fadc9c53be8/users/883ea5aef1424b6487a4bb1ba56ad492/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:33.278963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03108bad-9855-409b-8ab1-1caaece1635f admin admin] RBAC: Authorizing `identity:create_grant(project_id=cabea523b9aa4ee7a5fb7fadc9c53be8, user_id=883ea5aef1424b6487a4bb1ba56ad492, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:33.281553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03108bad-9855-409b-8ab1-1caaece1635f admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:33.292400 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1313/2637] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:27:33 2022] PUT /identity/v3/projects/cabea523b9aa4ee7a5fb7fadc9c53be8/users/883ea5aef1424b6487a4bb1ba56ad492/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:27:33.302371 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84af8615-ae46-4722-9711-30efa11ba0c7 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:33.302662 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84af8615-ae46-4722-9711-30efa11ba0c7 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:33.302939 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84af8615-ae46-4722-9711-30efa11ba0c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:33.331062 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-84af8615-ae46-4722-9711-30efa11ba0c7 None None] MFA Rules not processed for user `883ea5aef1424b6487a4bb1ba56ad492`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:27:33.332170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-84af8615-ae46-4722-9711-30efa11ba0c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:33.399885 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1325/2638] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:27:33 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:27:33.411088 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03108bad-9855-409b-8ab1-1caaece1635f admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:33.412815 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-03108bad-9855-409b-8ab1-1caaece1635f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:33.458644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:27:33.471734 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03108bad-9855-409b-8ab1-1caaece1635f admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:33.475340 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9390e621-eba7-4c64-8395-e7bd6f7dd9b8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:33.475934 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9390e621-eba7-4c64-8395-e7bd6f7dd9b8 service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:33.476208 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9390e621-eba7-4c64-8395-e7bd6f7dd9b8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:33.476477 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9390e621-eba7-4c64-8395-e7bd6f7dd9b8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:33.485178 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9390e621-eba7-4c64-8395-e7bd6f7dd9b8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:33.486621 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9390e621-eba7-4c64-8395-e7bd6f7dd9b8 service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:33.495833 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1314/2639] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:27:33 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:27:33.646283 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84af8615-ae46-4722-9711-30efa11ba0c7 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:33.655742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84af8615-ae46-4722-9711-30efa11ba0c7 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:33.659238 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5652b2-32ee-46e4-bdb5-27723045c82e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:33.659834 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5652b2-32ee-46e4-bdb5-27723045c82e service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:33.660108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5652b2-32ee-46e4-bdb5-27723045c82e service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:33.660374 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5652b2-32ee-46e4-bdb5-27723045c82e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:33.720376 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5652b2-32ee-46e4-bdb5-27723045c82e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:33.721953 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5652b2-32ee-46e4-bdb5-27723045c82e service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:33.731634 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1326/2640] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:27:33 2022] GET /identity/v3/auth/tokens => generated 3284 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:27:41.063111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9390e621-eba7-4c64-8395-e7bd6f7dd9b8 service cinder] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:41.123370 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9390e621-eba7-4c64-8395-e7bd6f7dd9b8 service cinder] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:41.127842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298fc61d-3a91-42f2-ab9c-c273c69f8938 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:41.128463 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298fc61d-3a91-42f2-ab9c-c273c69f8938 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:41.128740 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298fc61d-3a91-42f2-ab9c-c273c69f8938 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:41.129042 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298fc61d-3a91-42f2-ab9c-c273c69f8938 admin admin] PATH_INFO: `/v3/users/9cc9368ddeea4dc997075d1024ec518d` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:41.130180 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298fc61d-3a91-42f2-ab9c-c273c69f8938 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9cc9368ddeea4dc997075d1024ec518d)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:41.132061 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298fc61d-3a91-42f2-ab9c-c273c69f8938 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:41.177374 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-298fc61d-3a91-42f2-ab9c-c273c69f8938 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9cc9368ddeea4dc997075d1024ec518d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:41.184097 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-298fc61d-3a91-42f2-ab9c-c273c69f8938 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9cc9368ddeea4dc997075d1024ec518d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:41.184765 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-298fc61d-3a91-42f2-ab9c-c273c69f8938 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9cc9368ddeea4dc997075d1024ec518d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:41.192425 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-298fc61d-3a91-42f2-ab9c-c273c69f8938 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9cc9368ddeea4dc997075d1024ec518d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:41.204280 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-298fc61d-3a91-42f2-ab9c-c273c69f8938 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9cc9368ddeea4dc997075d1024ec518d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:41.210039 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1315/2641] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:27:41 2022] DELETE /identity/v3/users/9cc9368ddeea4dc997075d1024ec518d => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:27:41.214468 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5805583c-9c06-4b3d-8916-0942c2bd96c8 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:41.214927 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5805583c-9c06-4b3d-8916-0942c2bd96c8 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:41.215047 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5805583c-9c06-4b3d-8916-0942c2bd96c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:41.234001 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-5805583c-9c06-4b3d-8916-0942c2bd96c8 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:27:41.235182 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-5805583c-9c06-4b3d-8916-0942c2bd96c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:41.273338 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=100 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:27:41.292483 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1327/2642] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:27:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:27:41.302838 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298fc61d-3a91-42f2-ab9c-c273c69f8938 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:41.304329 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-298fc61d-3a91-42f2-ab9c-c273c69f8938 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:41.361021 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298fc61d-3a91-42f2-ab9c-c273c69f8938 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:41.365056 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee782a40-cf52-45fc-b309-ba30db1ee7eb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:41.365661 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee782a40-cf52-45fc-b309-ba30db1ee7eb service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:41.365968 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee782a40-cf52-45fc-b309-ba30db1ee7eb service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:41.366241 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee782a40-cf52-45fc-b309-ba30db1ee7eb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:41.375134 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee782a40-cf52-45fc-b309-ba30db1ee7eb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:41.376546 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee782a40-cf52-45fc-b309-ba30db1ee7eb service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:41.385956 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1316/2643] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:27:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:27:41.650769 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5805583c-9c06-4b3d-8916-0942c2bd96c8 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:41.652311 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-5805583c-9c06-4b3d-8916-0942c2bd96c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:41.664720 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5805583c-9c06-4b3d-8916-0942c2bd96c8 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:41.670477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af97fee1-ca67-436e-9570-51cfc7da798d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:41.671127 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af97fee1-ca67-436e-9570-51cfc7da798d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:41.671404 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af97fee1-ca67-436e-9570-51cfc7da798d admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:41.671681 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af97fee1-ca67-436e-9570-51cfc7da798d admin admin] PATH_INFO: `/v3/projects/cb019f622c4b488197b720549a9ffa73` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:41.672879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af97fee1-ca67-436e-9570-51cfc7da798d admin admin] RBAC: Authorizing `identity:delete_project(project_id=cb019f622c4b488197b720549a9ffa73)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:41.674895 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af97fee1-ca67-436e-9570-51cfc7da798d admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:41.727759 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-af97fee1-ca67-436e-9570-51cfc7da798d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cb019f622c4b488197b720549a9ffa73', 'request_id': 'req-af97fee1-ca67-436e-9570-51cfc7da798d'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:41.733256 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-af97fee1-ca67-436e-9570-51cfc7da798d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cb019f622c4b488197b720549a9ffa73', 'request_id': 'req-af97fee1-ca67-436e-9570-51cfc7da798d'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:41.739368 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-af97fee1-ca67-436e-9570-51cfc7da798d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cb019f622c4b488197b720549a9ffa73', 'request_id': 'req-af97fee1-ca67-436e-9570-51cfc7da798d'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:41.751876 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-af97fee1-ca67-436e-9570-51cfc7da798d admin admin] The token cache is being invalidate because project cb019f622c4b488197b720549a9ffa73 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:27:41.752167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-af97fee1-ca67-436e-9570-51cfc7da798d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:41.752925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-af97fee1-ca67-436e-9570-51cfc7da798d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:41.754951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1328/2644] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:27:41 2022] DELETE /identity/v3/projects/cb019f622c4b488197b720549a9ffa73 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:27:41.758902 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee782a40-cf52-45fc-b309-ba30db1ee7eb service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:41.770010 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee782a40-cf52-45fc-b309-ba30db1ee7eb service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:41.773796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:27:41.833846 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7456c03-6ddc-4c37-ae55-702b9818e038 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:41.834497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7456c03-6ddc-4c37-ae55-702b9818e038 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:41.834774 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7456c03-6ddc-4c37-ae55-702b9818e038 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:41.835072 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7456c03-6ddc-4c37-ae55-702b9818e038 admin admin] PATH_INFO: `/v3/users/883ea5aef1424b6487a4bb1ba56ad492` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:41.836176 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7456c03-6ddc-4c37-ae55-702b9818e038 admin admin] RBAC: Authorizing `identity:delete_user(user_id=883ea5aef1424b6487a4bb1ba56ad492)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:41.838089 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7456c03-6ddc-4c37-ae55-702b9818e038 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:41.882688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-f7456c03-6ddc-4c37-ae55-702b9818e038 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '883ea5aef1424b6487a4bb1ba56ad492'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:41.889413 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-f7456c03-6ddc-4c37-ae55-702b9818e038 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '883ea5aef1424b6487a4bb1ba56ad492'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:41.890086 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-f7456c03-6ddc-4c37-ae55-702b9818e038 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '883ea5aef1424b6487a4bb1ba56ad492'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:41.897847 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-f7456c03-6ddc-4c37-ae55-702b9818e038 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '883ea5aef1424b6487a4bb1ba56ad492'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:41.910142 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-f7456c03-6ddc-4c37-ae55-702b9818e038 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '883ea5aef1424b6487a4bb1ba56ad492'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:41.916193 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1317/2645] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:27:41 2022] DELETE /identity/v3/users/883ea5aef1424b6487a4bb1ba56ad492 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:27:42.198335 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af97fee1-ca67-436e-9570-51cfc7da798d admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:42.199788 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-af97fee1-ca67-436e-9570-51cfc7da798d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:42.213148 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af97fee1-ca67-436e-9570-51cfc7da798d admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:42.268708 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abeee37-c3e0-4a77-a2fc-aeebbe477acd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:42.269409 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abeee37-c3e0-4a77-a2fc-aeebbe477acd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:42.269620 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abeee37-c3e0-4a77-a2fc-aeebbe477acd admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:42.269926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abeee37-c3e0-4a77-a2fc-aeebbe477acd admin admin] PATH_INFO: `/v3/projects/cabea523b9aa4ee7a5fb7fadc9c53be8` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:42.271020 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abeee37-c3e0-4a77-a2fc-aeebbe477acd admin admin] RBAC: Authorizing `identity:delete_project(project_id=cabea523b9aa4ee7a5fb7fadc9c53be8)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:42.272904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abeee37-c3e0-4a77-a2fc-aeebbe477acd admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:42.319342 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-9abeee37-c3e0-4a77-a2fc-aeebbe477acd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cabea523b9aa4ee7a5fb7fadc9c53be8', 'request_id': 'req-9abeee37-c3e0-4a77-a2fc-aeebbe477acd'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:42.324745 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-9abeee37-c3e0-4a77-a2fc-aeebbe477acd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cabea523b9aa4ee7a5fb7fadc9c53be8', 'request_id': 'req-9abeee37-c3e0-4a77-a2fc-aeebbe477acd'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:42.330676 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-9abeee37-c3e0-4a77-a2fc-aeebbe477acd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cabea523b9aa4ee7a5fb7fadc9c53be8', 'request_id': 'req-9abeee37-c3e0-4a77-a2fc-aeebbe477acd'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:42.338391 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:27:42.344295 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-9abeee37-c3e0-4a77-a2fc-aeebbe477acd admin admin] The token cache is being invalidate because project cabea523b9aa4ee7a5fb7fadc9c53be8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:27:42.344805 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-9abeee37-c3e0-4a77-a2fc-aeebbe477acd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:42.345864 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-9abeee37-c3e0-4a77-a2fc-aeebbe477acd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:42.347476 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1329/2646] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:27:42 2022] DELETE /identity/v3/projects/cabea523b9aa4ee7a5fb7fadc9c53be8 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:27:42.352298 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dae6e6-839a-4583-8891-50c4f8dd4594 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:42.352783 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dae6e6-839a-4583-8891-50c4f8dd4594 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:42.352892 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dae6e6-839a-4583-8891-50c4f8dd4594 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:42.372646 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-39dae6e6-839a-4583-8891-50c4f8dd4594 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:27:42.373889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-39dae6e6-839a-4583-8891-50c4f8dd4594 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:42.432401 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1318/2647] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:27:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:27:42.440108 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda5a512-3d55-4a14-bd2d-4b10235acffb None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:42.440442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda5a512-3d55-4a14-bd2d-4b10235acffb None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:42.440662 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda5a512-3d55-4a14-bd2d-4b10235acffb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:42.460252 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-fda5a512-3d55-4a14-bd2d-4b10235acffb None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:27:42.461426 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-fda5a512-3d55-4a14-bd2d-4b10235acffb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:42.469074 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1330/2648] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:27:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:27:42.473700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39dae6e6-839a-4583-8891-50c4f8dd4594 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:42.483436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39dae6e6-839a-4583-8891-50c4f8dd4594 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:42.487963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fff256-924f-4378-9dcb-7b8088d7b4e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:42.488551 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fff256-924f-4378-9dcb-7b8088d7b4e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:42.488833 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fff256-924f-4378-9dcb-7b8088d7b4e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:42.489109 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fff256-924f-4378-9dcb-7b8088d7b4e8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:42.489802 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23fff256-924f-4378-9dcb-7b8088d7b4e8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:27:42.490109 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23fff256-924f-4378-9dcb-7b8088d7b4e8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:42.491697 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23fff256-924f-4378-9dcb-7b8088d7b4e8 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:42.503100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1319/2649] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:27:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:27:42.507761 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda5a512-3d55-4a14-bd2d-4b10235acffb None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:42.516658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda5a512-3d55-4a14-bd2d-4b10235acffb None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:42.521864 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f9c510-084e-43f1-ab30-df72f30c2eca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:42.522441 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f9c510-084e-43f1-ab30-df72f30c2eca admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:42.522742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f9c510-084e-43f1-ab30-df72f30c2eca admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:42.523013 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f9c510-084e-43f1-ab30-df72f30c2eca admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:42.523624 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f9c510-084e-43f1-ab30-df72f30c2eca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:42.525497 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f9c510-084e-43f1-ab30-df72f30c2eca admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:42.551600 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1331/2650] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:27:42 2022] POST /identity/v3/projects => generated 383 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:27:42.555844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fff256-924f-4378-9dcb-7b8088d7b4e8 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:42.565619 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fff256-924f-4378-9dcb-7b8088d7b4e8 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:42.574951 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:27:42.624742 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6fd423-f1f1-4142-876e-da5693085622 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:42.625339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6fd423-f1f1-4142-876e-da5693085622 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:42.625616 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6fd423-f1f1-4142-876e-da5693085622 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:42.625922 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6fd423-f1f1-4142-876e-da5693085622 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:42.626537 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6fd423-f1f1-4142-876e-da5693085622 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:42.628420 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6fd423-f1f1-4142-876e-da5693085622 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:42.649873 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1320/2651] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:27:42 2022] POST /identity/v3/users => generated 292 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:27:42.653973 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f9c510-084e-43f1-ab30-df72f30c2eca admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:42.664479 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f9c510-084e-43f1-ab30-df72f30c2eca admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:42.669326 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd8cd15-524d-4604-b61c-f5c7bc5090a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:42.669962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd8cd15-524d-4604-b61c-f5c7bc5090a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:42.670240 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd8cd15-524d-4604-b61c-f5c7bc5090a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:42.670541 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd8cd15-524d-4604-b61c-f5c7bc5090a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:42.671348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd8cd15-524d-4604-b61c-f5c7bc5090a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:42.672924 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd8cd15-524d-4604-b61c-f5c7bc5090a1 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:42.682596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1332/2652] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:27:42 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:27:42.686373 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6fd423-f1f1-4142-876e-da5693085622 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:42.696832 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6fd423-f1f1-4142-876e-da5693085622 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:42.701287 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3da09aa-c930-4f3f-be15-5958bb16d147 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:42.701905 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3da09aa-c930-4f3f-be15-5958bb16d147 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:42.702190 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3da09aa-c930-4f3f-be15-5958bb16d147 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:42.702465 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3da09aa-c930-4f3f-be15-5958bb16d147 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:42.703104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3da09aa-c930-4f3f-be15-5958bb16d147 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:42.704820 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3da09aa-c930-4f3f-be15-5958bb16d147 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:42.715966 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1321/2653] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:27:42 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:27:42.720055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd8cd15-524d-4604-b61c-f5c7bc5090a1 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:42.730382 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd8cd15-524d-4604-b61c-f5c7bc5090a1 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:42.734964 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcaa76f-e0af-47f3-b356-f18559721d50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:42.735627 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcaa76f-e0af-47f3-b356-f18559721d50 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:42.735905 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcaa76f-e0af-47f3-b356-f18559721d50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:42.736197 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcaa76f-e0af-47f3-b356-f18559721d50 admin admin] PATH_INFO: `/v3/projects/4893e67386394cf39813dee5fb9acc07/users/27f52991680e401c9a4bca56c5a79b83/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:42.750846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fcaa76f-e0af-47f3-b356-f18559721d50 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4893e67386394cf39813dee5fb9acc07, user_id=27f52991680e401c9a4bca56c5a79b83, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:42.753458 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fcaa76f-e0af-47f3-b356-f18559721d50 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:42.766636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1333/2654] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:27:42 2022] PUT /identity/v3/projects/4893e67386394cf39813dee5fb9acc07/users/27f52991680e401c9a4bca56c5a79b83/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:27:42.778510 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffa71af-9915-4d9e-9dd0-97473d124149 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:42.778795 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffa71af-9915-4d9e-9dd0-97473d124149 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:42.779071 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffa71af-9915-4d9e-9dd0-97473d124149 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:42.807378 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-7ffa71af-9915-4d9e-9dd0-97473d124149 None None] MFA Rules not processed for user `27f52991680e401c9a4bca56c5a79b83`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:27:42.808608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-7ffa71af-9915-4d9e-9dd0-97473d124149 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:42.871859 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1322/2655] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:27:42 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:27:42.876300 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcaa76f-e0af-47f3-b356-f18559721d50 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:42.939967 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcaa76f-e0af-47f3-b356-f18559721d50 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:42.945224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c1cf48-2384-44e7-a024-47edf60830a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:42.945857 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c1cf48-2384-44e7-a024-47edf60830a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:42.946141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c1cf48-2384-44e7-a024-47edf60830a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:42.946413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c1cf48-2384-44e7-a024-47edf60830a8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:42.947095 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c1cf48-2384-44e7-a024-47edf60830a8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:42.948916 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c1cf48-2384-44e7-a024-47edf60830a8 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:42.975315 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1334/2656] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:27:42 2022] POST /identity/v3/projects => generated 383 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:27:42.979266 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffa71af-9915-4d9e-9dd0-97473d124149 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:42.990257 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffa71af-9915-4d9e-9dd0-97473d124149 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:43.044690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9118d5e9-d9c3-4644-b875-f95b684ea195 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:43.045291 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9118d5e9-d9c3-4644-b875-f95b684ea195 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:43.045571 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9118d5e9-d9c3-4644-b875-f95b684ea195 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:43.045877 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9118d5e9-d9c3-4644-b875-f95b684ea195 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:43.046496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9118d5e9-d9c3-4644-b875-f95b684ea195 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:43.048391 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9118d5e9-d9c3-4644-b875-f95b684ea195 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:43.070125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1323/2657] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:27:42 2022] POST /identity/v3/users => generated 298 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:27:43.073847 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c1cf48-2384-44e7-a024-47edf60830a8 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:43.084378 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c1cf48-2384-44e7-a024-47edf60830a8 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:43.089103 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee86e47-f97b-46fb-a832-0734b612b2d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:43.089694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee86e47-f97b-46fb-a832-0734b612b2d2 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:43.090000 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee86e47-f97b-46fb-a832-0734b612b2d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:43.090271 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee86e47-f97b-46fb-a832-0734b612b2d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:43.090903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee86e47-f97b-46fb-a832-0734b612b2d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:43.092470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee86e47-f97b-46fb-a832-0734b612b2d2 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:43.103352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1335/2658] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:27:43 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:27:43.107277 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9118d5e9-d9c3-4644-b875-f95b684ea195 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:43.117013 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9118d5e9-d9c3-4644-b875-f95b684ea195 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:43.121537 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66defa7e-1f9f-47d1-9f68-22db6cc46fbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:43.122210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66defa7e-1f9f-47d1-9f68-22db6cc46fbb admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:43.122488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66defa7e-1f9f-47d1-9f68-22db6cc46fbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:43.122764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66defa7e-1f9f-47d1-9f68-22db6cc46fbb admin admin] PATH_INFO: `/v3/projects/c977ddf6660b495491868fbce80717dd/users/e4776751774147b9a9695ded704c68d3/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:43.131769 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:27:43.140063 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66defa7e-1f9f-47d1-9f68-22db6cc46fbb admin admin] RBAC: Authorizing `identity:create_grant(project_id=c977ddf6660b495491868fbce80717dd, user_id=e4776751774147b9a9695ded704c68d3, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:43.142805 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66defa7e-1f9f-47d1-9f68-22db6cc46fbb admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:43.156214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1324/2659] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:27:43 2022] PUT /identity/v3/projects/c977ddf6660b495491868fbce80717dd/users/e4776751774147b9a9695ded704c68d3/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:27:43.160787 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee86e47-f97b-46fb-a832-0734b612b2d2 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:43.171109 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee86e47-f97b-46fb-a832-0734b612b2d2 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:43.234298 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b03338-2063-4c24-8ae9-d7efa338eb24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:43.235032 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b03338-2063-4c24-8ae9-d7efa338eb24 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:43.235317 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b03338-2063-4c24-8ae9-d7efa338eb24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:43.235594 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b03338-2063-4c24-8ae9-d7efa338eb24 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:43.236235 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b03338-2063-4c24-8ae9-d7efa338eb24 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:43.237854 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b03338-2063-4c24-8ae9-d7efa338eb24 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:43.251094 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1336/2660] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:27:43 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:27:43.252688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66defa7e-1f9f-47d1-9f68-22db6cc46fbb admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:43.263592 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66defa7e-1f9f-47d1-9f68-22db6cc46fbb admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:43.268317 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f81ec2-c740-45b5-821c-b1f8d4a40800 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:43.268998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f81ec2-c740-45b5-821c-b1f8d4a40800 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:43.269828 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f81ec2-c740-45b5-821c-b1f8d4a40800 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:43.270109 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f81ec2-c740-45b5-821c-b1f8d4a40800 admin admin] PATH_INFO: `/v3/projects/c977ddf6660b495491868fbce80717dd/users/e4776751774147b9a9695ded704c68d3/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:43.272205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f81ec2-c740-45b5-821c-b1f8d4a40800 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c977ddf6660b495491868fbce80717dd, user_id=e4776751774147b9a9695ded704c68d3, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:43.274668 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f81ec2-c740-45b5-821c-b1f8d4a40800 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:43.285827 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1325/2661] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:27:43 2022] PUT /identity/v3/projects/c977ddf6660b495491868fbce80717dd/users/e4776751774147b9a9695ded704c68d3/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:27:43.296265 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9539ee-71b0-4d55-8b9f-0cf0d18331f9 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:43.296754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9539ee-71b0-4d55-8b9f-0cf0d18331f9 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:43.296858 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9539ee-71b0-4d55-8b9f-0cf0d18331f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:43.305692 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:27:43.333500 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-8f9539ee-71b0-4d55-8b9f-0cf0d18331f9 None None] MFA Rules not processed for user `e4776751774147b9a9695ded704c68d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:27:43.334761 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-8f9539ee-71b0-4d55-8b9f-0cf0d18331f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:43.400510 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1337/2662] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:27:43 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:27:43.412046 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f81ec2-c740-45b5-821c-b1f8d4a40800 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:43.413769 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-d6f81ec2-c740-45b5-821c-b1f8d4a40800 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:43.470592 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f81ec2-c740-45b5-821c-b1f8d4a40800 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:43.474128 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c26626-195b-40e2-abe8-b17a7a855e36 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:43.474721 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c26626-195b-40e2-abe8-b17a7a855e36 service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:43.475024 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c26626-195b-40e2-abe8-b17a7a855e36 service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:43.475292 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c26626-195b-40e2-abe8-b17a7a855e36 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:43.484143 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c26626-195b-40e2-abe8-b17a7a855e36 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:43.485555 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c26626-195b-40e2-abe8-b17a7a855e36 service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:43.494856 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1326/2663] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:27:43 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:27:43.586411 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9539ee-71b0-4d55-8b9f-0cf0d18331f9 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:43.596105 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9539ee-71b0-4d55-8b9f-0cf0d18331f9 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:43.599676 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c56b6bc-f090-4e5e-bcf3-4b0f1d754852 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:43.600299 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c56b6bc-f090-4e5e-bcf3-4b0f1d754852 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:43.600577 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c56b6bc-f090-4e5e-bcf3-4b0f1d754852 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:43.600849 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c56b6bc-f090-4e5e-bcf3-4b0f1d754852 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:43.656871 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c56b6bc-f090-4e5e-bcf3-4b0f1d754852 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:43.658337 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c56b6bc-f090-4e5e-bcf3-4b0f1d754852 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:43.668433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1338/2664] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:27:43 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:27:43.935638 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c26626-195b-40e2-abe8-b17a7a855e36 service cinder] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:43.999848 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c26626-195b-40e2-abe8-b17a7a855e36 service cinder] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:44.004434 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9181996e-aafc-4b2e-b076-1caeecfb9a4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:44.005057 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9181996e-aafc-4b2e-b076-1caeecfb9a4c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:44.005332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9181996e-aafc-4b2e-b076-1caeecfb9a4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:44.005602 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9181996e-aafc-4b2e-b076-1caeecfb9a4c admin admin] PATH_INFO: `/v3/users/27f52991680e401c9a4bca56c5a79b83` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:44.006789 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9181996e-aafc-4b2e-b076-1caeecfb9a4c admin admin] RBAC: Authorizing `identity:delete_user(user_id=27f52991680e401c9a4bca56c5a79b83)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:44.008613 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9181996e-aafc-4b2e-b076-1caeecfb9a4c admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:44.054231 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-9181996e-aafc-4b2e-b076-1caeecfb9a4c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '27f52991680e401c9a4bca56c5a79b83'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:44.061762 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-9181996e-aafc-4b2e-b076-1caeecfb9a4c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '27f52991680e401c9a4bca56c5a79b83'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:44.062452 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-9181996e-aafc-4b2e-b076-1caeecfb9a4c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '27f52991680e401c9a4bca56c5a79b83'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:44.078837 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-9181996e-aafc-4b2e-b076-1caeecfb9a4c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '27f52991680e401c9a4bca56c5a79b83'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:44.090195 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:27:44.091478 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-9181996e-aafc-4b2e-b076-1caeecfb9a4c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '27f52991680e401c9a4bca56c5a79b83'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:44.099077 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1327/2665] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:27:43 2022] DELETE /identity/v3/users/27f52991680e401c9a4bca56c5a79b83 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:27:44.103723 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013d71e3-35dd-44f4-91c9-6f76ce404d56 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:44.104074 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013d71e3-35dd-44f4-91c9-6f76ce404d56 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:44.104447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013d71e3-35dd-44f4-91c9-6f76ce404d56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:44.124690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-013d71e3-35dd-44f4-91c9-6f76ce404d56 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:27:44.126053 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-013d71e3-35dd-44f4-91c9-6f76ce404d56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:44.306902 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1339/2666] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:27:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:27:44.317694 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9181996e-aafc-4b2e-b076-1caeecfb9a4c admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:44.319533 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-9181996e-aafc-4b2e-b076-1caeecfb9a4c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:44.381881 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9181996e-aafc-4b2e-b076-1caeecfb9a4c admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:44.385654 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-094e68f0-f776-45fc-aea8-e6008252111b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:44.386589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094e68f0-f776-45fc-aea8-e6008252111b service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:44.386728 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094e68f0-f776-45fc-aea8-e6008252111b service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:44.387025 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094e68f0-f776-45fc-aea8-e6008252111b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:44.397112 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-094e68f0-f776-45fc-aea8-e6008252111b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:44.398714 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-094e68f0-f776-45fc-aea8-e6008252111b service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:44.408983 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1328/2667] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:27:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:27:44.708757 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013d71e3-35dd-44f4-91c9-6f76ce404d56 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:44.710432 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-013d71e3-35dd-44f4-91c9-6f76ce404d56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:44.723337 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013d71e3-35dd-44f4-91c9-6f76ce404d56 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:44.728086 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c8c2c5-5299-4afa-949b-32839d6f2d9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:44.728711 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c8c2c5-5299-4afa-949b-32839d6f2d9d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:44.729009 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c8c2c5-5299-4afa-949b-32839d6f2d9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:44.729348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c8c2c5-5299-4afa-949b-32839d6f2d9d admin admin] PATH_INFO: `/v3/projects/4893e67386394cf39813dee5fb9acc07` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:44.730463 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c8c2c5-5299-4afa-949b-32839d6f2d9d admin admin] RBAC: Authorizing `identity:delete_project(project_id=4893e67386394cf39813dee5fb9acc07)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:44.736766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c8c2c5-5299-4afa-949b-32839d6f2d9d admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:44.765175 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:27:44.791013 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-05c8c2c5-5299-4afa-949b-32839d6f2d9d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4893e67386394cf39813dee5fb9acc07', 'request_id': 'req-05c8c2c5-5299-4afa-949b-32839d6f2d9d'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:44.796428 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-05c8c2c5-5299-4afa-949b-32839d6f2d9d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4893e67386394cf39813dee5fb9acc07', 'request_id': 'req-05c8c2c5-5299-4afa-949b-32839d6f2d9d'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:44.804075 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-05c8c2c5-5299-4afa-949b-32839d6f2d9d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4893e67386394cf39813dee5fb9acc07', 'request_id': 'req-05c8c2c5-5299-4afa-949b-32839d6f2d9d'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:44.817525 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-05c8c2c5-5299-4afa-949b-32839d6f2d9d admin admin] The token cache is being invalidate because project 4893e67386394cf39813dee5fb9acc07 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:27:44.817844 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-05c8c2c5-5299-4afa-949b-32839d6f2d9d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:44.818460 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-05c8c2c5-5299-4afa-949b-32839d6f2d9d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:44.819941 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1340/2668] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:27:44 2022] DELETE /identity/v3/projects/4893e67386394cf39813dee5fb9acc07 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:27:44.823979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-094e68f0-f776-45fc-aea8-e6008252111b service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:44.834286 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-094e68f0-f776-45fc-aea8-e6008252111b service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:44.888959 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bb6c91-5568-4665-b581-bddd3c6b18b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:44.889579 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bb6c91-5568-4665-b581-bddd3c6b18b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:44.889900 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bb6c91-5568-4665-b581-bddd3c6b18b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:44.890180 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bb6c91-5568-4665-b581-bddd3c6b18b7 admin admin] PATH_INFO: `/v3/users/e4776751774147b9a9695ded704c68d3` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:44.891284 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15bb6c91-5568-4665-b581-bddd3c6b18b7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e4776751774147b9a9695ded704c68d3)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:44.893183 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15bb6c91-5568-4665-b581-bddd3c6b18b7 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:44.937015 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-15bb6c91-5568-4665-b581-bddd3c6b18b7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e4776751774147b9a9695ded704c68d3'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:44.944478 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-15bb6c91-5568-4665-b581-bddd3c6b18b7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e4776751774147b9a9695ded704c68d3'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:44.945131 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-15bb6c91-5568-4665-b581-bddd3c6b18b7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e4776751774147b9a9695ded704c68d3'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:44.951932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-15bb6c91-5568-4665-b581-bddd3c6b18b7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e4776751774147b9a9695ded704c68d3'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:44.965219 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-15bb6c91-5568-4665-b581-bddd3c6b18b7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e4776751774147b9a9695ded704c68d3'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:44.971869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1329/2669] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:27:44 2022] DELETE /identity/v3/users/e4776751774147b9a9695ded704c68d3 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:27:45.234630 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c8c2c5-5299-4afa-949b-32839d6f2d9d admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:45.236085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-05c8c2c5-5299-4afa-949b-32839d6f2d9d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:45.249211 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c8c2c5-5299-4afa-949b-32839d6f2d9d admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:45.304347 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e8120a-694c-4385-b452-c6dec74847e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:45.304971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e8120a-694c-4385-b452-c6dec74847e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:45.305249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e8120a-694c-4385-b452-c6dec74847e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:45.305520 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e8120a-694c-4385-b452-c6dec74847e1 admin admin] PATH_INFO: `/v3/projects/c977ddf6660b495491868fbce80717dd` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:45.306646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e8120a-694c-4385-b452-c6dec74847e1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c977ddf6660b495491868fbce80717dd)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:45.308552 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e8120a-694c-4385-b452-c6dec74847e1 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:45.354075 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-45e8120a-694c-4385-b452-c6dec74847e1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c977ddf6660b495491868fbce80717dd', 'request_id': 'req-45e8120a-694c-4385-b452-c6dec74847e1'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:45.359426 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-45e8120a-694c-4385-b452-c6dec74847e1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c977ddf6660b495491868fbce80717dd', 'request_id': 'req-45e8120a-694c-4385-b452-c6dec74847e1'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:45.366018 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-45e8120a-694c-4385-b452-c6dec74847e1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c977ddf6660b495491868fbce80717dd', 'request_id': 'req-45e8120a-694c-4385-b452-c6dec74847e1'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:45.377937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-45e8120a-694c-4385-b452-c6dec74847e1 admin admin] The token cache is being invalidate because project c977ddf6660b495491868fbce80717dd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:27:45.378223 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-45e8120a-694c-4385-b452-c6dec74847e1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:45.379552 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-45e8120a-694c-4385-b452-c6dec74847e1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:45.381078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1341/2670] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:27:45 2022] DELETE /identity/v3/projects/c977ddf6660b495491868fbce80717dd => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:27:45.385896 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44ede02-21a6-49fe-a1a9-a86297190f4e None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:45.386188 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44ede02-21a6-49fe-a1a9-a86297190f4e None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:45.386467 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44ede02-21a6-49fe-a1a9-a86297190f4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:45.406688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-a44ede02-21a6-49fe-a1a9-a86297190f4e None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:27:45.407900 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-a44ede02-21a6-49fe-a1a9-a86297190f4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:45.420751 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:27:45.469016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1330/2671] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:27:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:27:45.478351 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699b6b40-229d-45aa-a790-065ee55f61a5 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:45.478629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699b6b40-229d-45aa-a790-065ee55f61a5 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:45.478898 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699b6b40-229d-45aa-a790-065ee55f61a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:45.498806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-699b6b40-229d-45aa-a790-065ee55f61a5 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:27:45.499971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-699b6b40-229d-45aa-a790-065ee55f61a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:45.508908 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1342/2672] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:27:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:27:45.513149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44ede02-21a6-49fe-a1a9-a86297190f4e None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:45.522937 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44ede02-21a6-49fe-a1a9-a86297190f4e None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:45.527546 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cfeaf2b-7745-4648-8f19-211f7be1ed3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:45.528145 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfeaf2b-7745-4648-8f19-211f7be1ed3c admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:45.528422 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfeaf2b-7745-4648-8f19-211f7be1ed3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:45.528693 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfeaf2b-7745-4648-8f19-211f7be1ed3c admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:45.529337 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cfeaf2b-7745-4648-8f19-211f7be1ed3c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:27:45.529668 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cfeaf2b-7745-4648-8f19-211f7be1ed3c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:45.531314 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cfeaf2b-7745-4648-8f19-211f7be1ed3c admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:45.544040 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1331/2673] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:27:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:27:45.548389 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699b6b40-229d-45aa-a790-065ee55f61a5 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:45.558088 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699b6b40-229d-45aa-a790-065ee55f61a5 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:45.562698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ef6211-1265-4a4b-8754-55dd0456b477 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:45.563284 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ef6211-1265-4a4b-8754-55dd0456b477 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:45.563561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ef6211-1265-4a4b-8754-55dd0456b477 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:45.563833 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ef6211-1265-4a4b-8754-55dd0456b477 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:45.564459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ef6211-1265-4a4b-8754-55dd0456b477 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:45.566369 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ef6211-1265-4a4b-8754-55dd0456b477 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:45.594516 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1343/2674] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:27:45 2022] POST /identity/v3/projects => generated 373 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:27:45.599650 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cfeaf2b-7745-4648-8f19-211f7be1ed3c admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:45.610218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cfeaf2b-7745-4648-8f19-211f7be1ed3c admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:45.677648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e57be7-f110-4211-ade7-1f210371a477 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:45.678300 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e57be7-f110-4211-ade7-1f210371a477 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:45.678680 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e57be7-f110-4211-ade7-1f210371a477 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:45.678959 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e57be7-f110-4211-ade7-1f210371a477 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:45.679608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e57be7-f110-4211-ade7-1f210371a477 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:45.681538 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e57be7-f110-4211-ade7-1f210371a477 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:45.702570 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1332/2675] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:27:45 2022] POST /identity/v3/users => generated 287 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:27:45.706935 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ef6211-1265-4a4b-8754-55dd0456b477 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:45.717632 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ef6211-1265-4a4b-8754-55dd0456b477 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:45.722162 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2184ba94-0ecb-4bc5-8d2d-f25b590d339b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:45.722751 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2184ba94-0ecb-4bc5-8d2d-f25b590d339b admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:45.723028 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2184ba94-0ecb-4bc5-8d2d-f25b590d339b admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:45.723301 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2184ba94-0ecb-4bc5-8d2d-f25b590d339b admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:45.723936 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2184ba94-0ecb-4bc5-8d2d-f25b590d339b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:45.725534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2184ba94-0ecb-4bc5-8d2d-f25b590d339b admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:45.735591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1344/2676] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:27:45 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:27:45.739387 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e57be7-f110-4211-ade7-1f210371a477 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:45.749515 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e57be7-f110-4211-ade7-1f210371a477 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:45.754662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53518c7-b5c2-4c2e-bf25-53fb0050c345 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:45.755248 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53518c7-b5c2-4c2e-bf25-53fb0050c345 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:45.755527 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53518c7-b5c2-4c2e-bf25-53fb0050c345 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:45.755804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53518c7-b5c2-4c2e-bf25-53fb0050c345 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:45.756436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53518c7-b5c2-4c2e-bf25-53fb0050c345 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:45.758026 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53518c7-b5c2-4c2e-bf25-53fb0050c345 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:45.768526 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1333/2677] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:27:45 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:27:45.773794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2184ba94-0ecb-4bc5-8d2d-f25b590d339b admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:45.779225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=84,keystone:DELETE=13,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:27:45.785515 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2184ba94-0ecb-4bc5-8d2d-f25b590d339b admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:45.791183 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68b05ba-4f6f-4f67-adfd-d35cffb4ced7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:45.791822 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68b05ba-4f6f-4f67-adfd-d35cffb4ced7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:45.792094 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68b05ba-4f6f-4f67-adfd-d35cffb4ced7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:45.792366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68b05ba-4f6f-4f67-adfd-d35cffb4ced7 admin admin] PATH_INFO: `/v3/projects/429b10571c42414d8400f0b2c58772ac/users/d594652e50c94910a7ee682c459618dc/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:45.806496 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68b05ba-4f6f-4f67-adfd-d35cffb4ced7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=429b10571c42414d8400f0b2c58772ac, user_id=d594652e50c94910a7ee682c459618dc, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:45.809049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68b05ba-4f6f-4f67-adfd-d35cffb4ced7 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:45.821344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1345/2678] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:27:45 2022] PUT /identity/v3/projects/429b10571c42414d8400f0b2c58772ac/users/d594652e50c94910a7ee682c459618dc/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:27:45.831587 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a212d120-5a78-4f3b-94dd-caaa8d8dedeb None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:45.832613 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a212d120-5a78-4f3b-94dd-caaa8d8dedeb None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:45.832721 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a212d120-5a78-4f3b-94dd-caaa8d8dedeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:45.860002 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-a212d120-5a78-4f3b-94dd-caaa8d8dedeb None None] MFA Rules not processed for user `d594652e50c94910a7ee682c459618dc`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:27:45.861262 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-a212d120-5a78-4f3b-94dd-caaa8d8dedeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:45.923031 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1334/2679] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:27:45 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:27:45.934926 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68b05ba-4f6f-4f67-adfd-d35cffb4ced7 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:45.936597 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a68b05ba-4f6f-4f67-adfd-d35cffb4ced7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:45.990736 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68b05ba-4f6f-4f67-adfd-d35cffb4ced7 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:45.994222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2201fdc7-c0b3-4b6b-991d-ccd1b483a1a0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:45.994812 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2201fdc7-c0b3-4b6b-991d-ccd1b483a1a0 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:45.995085 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2201fdc7-c0b3-4b6b-991d-ccd1b483a1a0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:45.995353 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2201fdc7-c0b3-4b6b-991d-ccd1b483a1a0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:46.003307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2201fdc7-c0b3-4b6b-991d-ccd1b483a1a0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:46.005354 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2201fdc7-c0b3-4b6b-991d-ccd1b483a1a0 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:46.013719 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1346/2680] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:27:45 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 79 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:27:46.991877 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a212d120-5a78-4f3b-94dd-caaa8d8dedeb None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:46.993449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-a212d120-5a78-4f3b-94dd-caaa8d8dedeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:47.011194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:27:47.061996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a212d120-5a78-4f3b-94dd-caaa8d8dedeb None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:47.065487 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416d3759-7a8e-4b4f-9674-f030bd58ca78 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:47.066667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416d3759-7a8e-4b4f-9674-f030bd58ca78 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:47.066943 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416d3759-7a8e-4b4f-9674-f030bd58ca78 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:47.067216 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416d3759-7a8e-4b4f-9674-f030bd58ca78 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:47.068959 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-416d3759-7a8e-4b4f-9674-f030bd58ca78 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:47.079058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416d3759-7a8e-4b4f-9674-f030bd58ca78 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:47.080543 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416d3759-7a8e-4b4f-9674-f030bd58ca78 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:47.089381 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1335/2681] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:27:46 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:27:47.161833 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2201fdc7-c0b3-4b6b-991d-ccd1b483a1a0 service cinder] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:47.171475 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2201fdc7-c0b3-4b6b-991d-ccd1b483a1a0 service cinder] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:47.174924 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82daa4a0-66e0-43f7-9abe-830bd508b62a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:47.175556 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82daa4a0-66e0-43f7-9abe-830bd508b62a service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:47.175833 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82daa4a0-66e0-43f7-9abe-830bd508b62a service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:47.176103 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82daa4a0-66e0-43f7-9abe-830bd508b62a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:47.177815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-82daa4a0-66e0-43f7-9abe-830bd508b62a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:47.247403 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82daa4a0-66e0-43f7-9abe-830bd508b62a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:47.248816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82daa4a0-66e0-43f7-9abe-830bd508b62a service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:47.260042 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1347/2682] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:27:47 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:27:49.042931 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416d3759-7a8e-4b4f-9674-f030bd58ca78 service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:49.112156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416d3759-7a8e-4b4f-9674-f030bd58ca78 service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:49.117585 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8860b297-e459-4045-80fa-f705deb61e0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:49.118280 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8860b297-e459-4045-80fa-f705deb61e0b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:49.118549 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8860b297-e459-4045-80fa-f705deb61e0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:49.118824 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8860b297-e459-4045-80fa-f705deb61e0b admin admin] PATH_INFO: `/v3/users/d594652e50c94910a7ee682c459618dc` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:49.119986 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8860b297-e459-4045-80fa-f705deb61e0b admin admin] RBAC: Authorizing `identity:delete_user(user_id=d594652e50c94910a7ee682c459618dc)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:49.121936 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8860b297-e459-4045-80fa-f705deb61e0b admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:49.175015 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8860b297-e459-4045-80fa-f705deb61e0b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd594652e50c94910a7ee682c459618dc'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:49.182697 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8860b297-e459-4045-80fa-f705deb61e0b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd594652e50c94910a7ee682c459618dc'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:49.183421 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8860b297-e459-4045-80fa-f705deb61e0b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd594652e50c94910a7ee682c459618dc'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:49.190965 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8860b297-e459-4045-80fa-f705deb61e0b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd594652e50c94910a7ee682c459618dc'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:49.206501 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8860b297-e459-4045-80fa-f705deb61e0b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd594652e50c94910a7ee682c459618dc'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:49.213421 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1336/2683] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:27:49 2022] DELETE /identity/v3/users/d594652e50c94910a7ee682c459618dc => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:27:49.218224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236d4e50-cdba-425b-a292-b466fe44abee None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:49.218749 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236d4e50-cdba-425b-a292-b466fe44abee None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:49.218923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236d4e50-cdba-425b-a292-b466fe44abee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:49.239399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-236d4e50-cdba-425b-a292-b466fe44abee None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:27:49.240713 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-236d4e50-cdba-425b-a292-b466fe44abee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:49.306014 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1348/2684] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:27:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:27:49.317087 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8860b297-e459-4045-80fa-f705deb61e0b admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:49.319087 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-8860b297-e459-4045-80fa-f705deb61e0b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:49.365783 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:27:49.387684 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8860b297-e459-4045-80fa-f705deb61e0b admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:49.392297 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d11f47-7b34-4c17-9b35-0df2e2db5161 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:49.392932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d11f47-7b34-4c17-9b35-0df2e2db5161 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:49.393257 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d11f47-7b34-4c17-9b35-0df2e2db5161 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:49.393474 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d11f47-7b34-4c17-9b35-0df2e2db5161 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:49.410894 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d11f47-7b34-4c17-9b35-0df2e2db5161 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:49.412454 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d11f47-7b34-4c17-9b35-0df2e2db5161 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:49.423433 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1337/2685] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:27:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:27:49.761208 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236d4e50-cdba-425b-a292-b466fe44abee None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:49.762747 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-236d4e50-cdba-425b-a292-b466fe44abee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:49.774510 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236d4e50-cdba-425b-a292-b466fe44abee None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:49.779239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba0234f-6c21-485e-acb2-82f8fd725fec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:49.779666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba0234f-6c21-485e-acb2-82f8fd725fec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:49.779942 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba0234f-6c21-485e-acb2-82f8fd725fec admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:49.780212 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba0234f-6c21-485e-acb2-82f8fd725fec admin admin] PATH_INFO: `/v3/projects/429b10571c42414d8400f0b2c58772ac` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:49.782249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba0234f-6c21-485e-acb2-82f8fd725fec admin admin] RBAC: Authorizing `identity:delete_project(project_id=429b10571c42414d8400f0b2c58772ac)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:49.784397 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba0234f-6c21-485e-acb2-82f8fd725fec admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:49.786961 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:27:49.834230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6ba0234f-6c21-485e-acb2-82f8fd725fec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '429b10571c42414d8400f0b2c58772ac', 'request_id': 'req-6ba0234f-6c21-485e-acb2-82f8fd725fec'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:49.839783 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6ba0234f-6c21-485e-acb2-82f8fd725fec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '429b10571c42414d8400f0b2c58772ac', 'request_id': 'req-6ba0234f-6c21-485e-acb2-82f8fd725fec'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:49.845960 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6ba0234f-6c21-485e-acb2-82f8fd725fec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '429b10571c42414d8400f0b2c58772ac', 'request_id': 'req-6ba0234f-6c21-485e-acb2-82f8fd725fec'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:49.858189 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6ba0234f-6c21-485e-acb2-82f8fd725fec admin admin] The token cache is being invalidate because project 429b10571c42414d8400f0b2c58772ac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:27:49.858474 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6ba0234f-6c21-485e-acb2-82f8fd725fec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:49.859088 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-6ba0234f-6c21-485e-acb2-82f8fd725fec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:27:49.860571 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1349/2686] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:27:49 2022] DELETE /identity/v3/projects/429b10571c42414d8400f0b2c58772ac => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:27:49.866173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d040e2b-db68-469f-9e13-74c5bd87bce8 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:49.866463 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d040e2b-db68-469f-9e13-74c5bd87bce8 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:49.866740 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d040e2b-db68-469f-9e13-74c5bd87bce8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:49.886565 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-0d040e2b-db68-469f-9e13-74c5bd87bce8 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:27:49.887764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0d040e2b-db68-469f-9e13-74c5bd87bce8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:49.947448 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1338/2687] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:27:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:27:49.954997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd74719-1157-4452-83e8-348fa07b3597 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:49.955282 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd74719-1157-4452-83e8-348fa07b3597 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:49.955557 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd74719-1157-4452-83e8-348fa07b3597 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:49.975458 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-ccd74719-1157-4452-83e8-348fa07b3597 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:27:49.976669 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ccd74719-1157-4452-83e8-348fa07b3597 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:49.984806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1350/2688] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:27:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:27:49.989590 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d040e2b-db68-469f-9e13-74c5bd87bce8 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:49.999291 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d040e2b-db68-469f-9e13-74c5bd87bce8 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:50.003904 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830bf634-6570-4640-8e54-baa5a37d2819 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:50.004507 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830bf634-6570-4640-8e54-baa5a37d2819 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:50.004781 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830bf634-6570-4640-8e54-baa5a37d2819 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:50.005054 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830bf634-6570-4640-8e54-baa5a37d2819 admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:50.005773 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830bf634-6570-4640-8e54-baa5a37d2819 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:27:50.006130 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830bf634-6570-4640-8e54-baa5a37d2819 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:50.007826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830bf634-6570-4640-8e54-baa5a37d2819 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:50.020968 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1339/2689] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:27:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:27:50.025228 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd74719-1157-4452-83e8-348fa07b3597 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:50.035437 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd74719-1157-4452-83e8-348fa07b3597 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:50.040464 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2492ba1-76db-47c5-8e78-5888b44c205e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:50.041325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2492ba1-76db-47c5-8e78-5888b44c205e admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:50.041596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2492ba1-76db-47c5-8e78-5888b44c205e admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:50.041904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2492ba1-76db-47c5-8e78-5888b44c205e admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:50.042541 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2492ba1-76db-47c5-8e78-5888b44c205e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:50.044466 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2492ba1-76db-47c5-8e78-5888b44c205e admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:50.074892 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1351/2690] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:27:50 2022] POST /identity/v3/projects => generated 375 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:27:50.078955 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830bf634-6570-4640-8e54-baa5a37d2819 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:50.089144 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830bf634-6570-4640-8e54-baa5a37d2819 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:50.150136 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0711771-acca-424c-893e-754dbe84b69f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:50.150740 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0711771-acca-424c-893e-754dbe84b69f admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:50.151023 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0711771-acca-424c-893e-754dbe84b69f admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:50.151299 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0711771-acca-424c-893e-754dbe84b69f admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:50.151924 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0711771-acca-424c-893e-754dbe84b69f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:50.153842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0711771-acca-424c-893e-754dbe84b69f admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:50.174906 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1340/2691] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:27:50 2022] POST /identity/v3/users => generated 288 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:27:50.179855 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2492ba1-76db-47c5-8e78-5888b44c205e admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:50.190332 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2492ba1-76db-47c5-8e78-5888b44c205e admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:50.195417 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d2ad44-90ea-4b25-af31-306c226212a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:50.196002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d2ad44-90ea-4b25-af31-306c226212a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:50.196281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d2ad44-90ea-4b25-af31-306c226212a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:50.196554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d2ad44-90ea-4b25-af31-306c226212a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:50.197190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d2ad44-90ea-4b25-af31-306c226212a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:50.198801 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d2ad44-90ea-4b25-af31-306c226212a9 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:50.209088 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1352/2692] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:27:50 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:27:50.212897 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0711771-acca-424c-893e-754dbe84b69f admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:50.222705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0711771-acca-424c-893e-754dbe84b69f admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:50.227290 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6647e60b-112d-4293-82b1-269931e23a50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:50.227878 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6647e60b-112d-4293-82b1-269931e23a50 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:50.228161 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6647e60b-112d-4293-82b1-269931e23a50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:50.228444 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6647e60b-112d-4293-82b1-269931e23a50 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:50.229072 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6647e60b-112d-4293-82b1-269931e23a50 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:50.230707 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6647e60b-112d-4293-82b1-269931e23a50 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:50.361324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1341/2693] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:27:50 2022] GET /identity/v3/roles => generated 1420 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:27:50.366424 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d2ad44-90ea-4b25-af31-306c226212a9 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:50.376287 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d2ad44-90ea-4b25-af31-306c226212a9 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:50.381597 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f931e18-ca08-49d1-b29f-8177a7eec86a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:50.382338 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f931e18-ca08-49d1-b29f-8177a7eec86a admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:50.382615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f931e18-ca08-49d1-b29f-8177a7eec86a admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:50.382888 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f931e18-ca08-49d1-b29f-8177a7eec86a admin admin] PATH_INFO: `/v3/projects/bb59f3d23e774836b006cce9b9d9b2b1/users/4e9d978a693a4e49a7fcf06a08d5af46/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:50.397167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f931e18-ca08-49d1-b29f-8177a7eec86a admin admin] RBAC: Authorizing `identity:create_grant(project_id=bb59f3d23e774836b006cce9b9d9b2b1, user_id=4e9d978a693a4e49a7fcf06a08d5af46, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:50.399785 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f931e18-ca08-49d1-b29f-8177a7eec86a admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:50.412674 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1353/2694] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:27:50 2022] PUT /identity/v3/projects/bb59f3d23e774836b006cce9b9d9b2b1/users/4e9d978a693a4e49a7fcf06a08d5af46/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:27:50.478761 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2cdd17-6a1f-44ea-9c9d-6d77ed8014b7 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:50.479046 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2cdd17-6a1f-44ea-9c9d-6d77ed8014b7 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:50.479323 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2cdd17-6a1f-44ea-9c9d-6d77ed8014b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:50.507158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-5b2cdd17-6a1f-44ea-9c9d-6d77ed8014b7 None None] MFA Rules not processed for user `4e9d978a693a4e49a7fcf06a08d5af46`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:27:50.508346 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-5b2cdd17-6a1f-44ea-9c9d-6d77ed8014b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:50.540009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:27:50.573491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1342/2695] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:27:50 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:27:50.585570 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f931e18-ca08-49d1-b29f-8177a7eec86a admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:27:50.587285 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6f931e18-ca08-49d1-b29f-8177a7eec86a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:27:50.652725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f931e18-ca08-49d1-b29f-8177a7eec86a admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:27:50.656303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d38424-501e-4036-b6b5-68a3e27f4281 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:27:50.656907 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d38424-501e-4036-b6b5-68a3e27f4281 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:27:50.657180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d38424-501e-4036-b6b5-68a3e27f4281 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:27:50.657448 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d38424-501e-4036-b6b5-68a3e27f4281 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:27:50.666260 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d38424-501e-4036-b6b5-68a3e27f4281 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:27:50.667680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d38424-501e-4036-b6b5-68a3e27f4281 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:27:50.677130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1354/2696] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:27:50 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:28:00.530014 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2cdd17-6a1f-44ea-9c9d-6d77ed8014b7 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:28:00.631969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2cdd17-6a1f-44ea-9c9d-6d77ed8014b7 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:28:00.636627 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b826a0f-4f49-4fb2-9da0-206a9321f6d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:28:00.637282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b826a0f-4f49-4fb2-9da0-206a9321f6d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:28:00.637558 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b826a0f-4f49-4fb2-9da0-206a9321f6d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:28:00.637865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b826a0f-4f49-4fb2-9da0-206a9321f6d4 admin admin] PATH_INFO: `/v3/users/4e9d978a693a4e49a7fcf06a08d5af46` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:28:00.638984 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b826a0f-4f49-4fb2-9da0-206a9321f6d4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4e9d978a693a4e49a7fcf06a08d5af46)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:28:00.640908 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b826a0f-4f49-4fb2-9da0-206a9321f6d4 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:28:00.672680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=77,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:28:00.689364 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8b826a0f-4f49-4fb2-9da0-206a9321f6d4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4e9d978a693a4e49a7fcf06a08d5af46'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:28:00.696396 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8b826a0f-4f49-4fb2-9da0-206a9321f6d4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4e9d978a693a4e49a7fcf06a08d5af46'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:28:00.697068 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8b826a0f-4f49-4fb2-9da0-206a9321f6d4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4e9d978a693a4e49a7fcf06a08d5af46'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:28:00.704382 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8b826a0f-4f49-4fb2-9da0-206a9321f6d4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4e9d978a693a4e49a7fcf06a08d5af46'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:28:00.716360 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-8b826a0f-4f49-4fb2-9da0-206a9321f6d4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4e9d978a693a4e49a7fcf06a08d5af46'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:28:00.722058 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1343/2697] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:28:00 2022] DELETE /identity/v3/users/4e9d978a693a4e49a7fcf06a08d5af46 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:28:00.727939 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d1ca39-6d1c-41a0-ba31-7d2b3adb7af0 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:28:00.728215 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d1ca39-6d1c-41a0-ba31-7d2b3adb7af0 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:28:00.728481 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d1ca39-6d1c-41a0-ba31-7d2b3adb7af0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:28:00.776105 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-49d1ca39-6d1c-41a0-ba31-7d2b3adb7af0 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:28:00.777296 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-49d1ca39-6d1c-41a0-ba31-7d2b3adb7af0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:28:00.834543 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1355/2698] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:28:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:28:00.844233 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b826a0f-4f49-4fb2-9da0-206a9321f6d4 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:28:00.845619 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-8b826a0f-4f49-4fb2-9da0-206a9321f6d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:28:00.863752 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:28:00.924970 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b826a0f-4f49-4fb2-9da0-206a9321f6d4 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:28:00.928558 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97ce442-3e5b-4971-8200-f23100835380 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:28:00.929155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97ce442-3e5b-4971-8200-f23100835380 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:28:00.929434 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97ce442-3e5b-4971-8200-f23100835380 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:28:00.929707 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97ce442-3e5b-4971-8200-f23100835380 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:28:00.938515 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97ce442-3e5b-4971-8200-f23100835380 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:28:00.939929 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97ce442-3e5b-4971-8200-f23100835380 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:28:00.949419 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1344/2699] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:28:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:28:01.206973 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d1ca39-6d1c-41a0-ba31-7d2b3adb7af0 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:28:01.209080 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-49d1ca39-6d1c-41a0-ba31-7d2b3adb7af0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:28:01.224682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d1ca39-6d1c-41a0-ba31-7d2b3adb7af0 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:28:01.229925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6ca772-218e-46bd-881a-c53f5aef6cc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:28:01.231590 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6ca772-218e-46bd-881a-c53f5aef6cc8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:28:01.231881 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6ca772-218e-46bd-881a-c53f5aef6cc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:28:01.232156 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6ca772-218e-46bd-881a-c53f5aef6cc8 admin admin] PATH_INFO: `/v3/projects/bb59f3d23e774836b006cce9b9d9b2b1` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:28:01.233373 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6ca772-218e-46bd-881a-c53f5aef6cc8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bb59f3d23e774836b006cce9b9d9b2b1)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:28:01.235211 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6ca772-218e-46bd-881a-c53f5aef6cc8 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:28:01.283162 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0f6ca772-218e-46bd-881a-c53f5aef6cc8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bb59f3d23e774836b006cce9b9d9b2b1', 'request_id': 'req-0f6ca772-218e-46bd-881a-c53f5aef6cc8'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:28:01.288809 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0f6ca772-218e-46bd-881a-c53f5aef6cc8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bb59f3d23e774836b006cce9b9d9b2b1', 'request_id': 'req-0f6ca772-218e-46bd-881a-c53f5aef6cc8'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:28:01.295205 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0f6ca772-218e-46bd-881a-c53f5aef6cc8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bb59f3d23e774836b006cce9b9d9b2b1', 'request_id': 'req-0f6ca772-218e-46bd-881a-c53f5aef6cc8'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:28:01.307269 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0f6ca772-218e-46bd-881a-c53f5aef6cc8 admin admin] The token cache is being invalidate because project bb59f3d23e774836b006cce9b9d9b2b1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:28:01.307558 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0f6ca772-218e-46bd-881a-c53f5aef6cc8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:28:01.308189 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-0f6ca772-218e-46bd-881a-c53f5aef6cc8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:28:01.309646 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1356/2700] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:28:01 2022] DELETE /identity/v3/projects/bb59f3d23e774836b006cce9b9d9b2b1 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:28:01.315053 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5c3530-093a-4baf-84fd-28388d9eafe0 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:28:01.315346 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5c3530-093a-4baf-84fd-28388d9eafe0 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:28:01.315626 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5c3530-093a-4baf-84fd-28388d9eafe0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:28:01.363990 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-1d5c3530-093a-4baf-84fd-28388d9eafe0 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:28:01.365223 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-1d5c3530-093a-4baf-84fd-28388d9eafe0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:28:01.422605 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1345/2701] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:28:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:28:01.430083 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5dc055-d302-440e-bf98-77e9350c7c82 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:28:01.430368 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5dc055-d302-440e-bf98-77e9350c7c82 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:28:01.430647 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5dc055-d302-440e-bf98-77e9350c7c82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:28:01.450211 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-6e5dc055-d302-440e-bf98-77e9350c7c82 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:28:01.451340 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6e5dc055-d302-440e-bf98-77e9350c7c82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:28:01.460117 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1357/2702] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:28:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:28:01.464109 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5c3530-093a-4baf-84fd-28388d9eafe0 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:28:01.473407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5c3530-093a-4baf-84fd-28388d9eafe0 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:28:01.477876 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c6b85d-0979-423c-bbd1-e8073a1910ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:28:01.478453 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c6b85d-0979-423c-bbd1-e8073a1910ec admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:28:01.478730 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c6b85d-0979-423c-bbd1-e8073a1910ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:28:01.479003 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c6b85d-0979-423c-bbd1-e8073a1910ec admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:28:01.479637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c6b85d-0979-423c-bbd1-e8073a1910ec admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:28:01.479936 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c6b85d-0979-423c-bbd1-e8073a1910ec admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:28:01.481488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c6b85d-0979-423c-bbd1-e8073a1910ec admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:28:01.492592 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1346/2703] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:28:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:28:01.497117 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5dc055-d302-440e-bf98-77e9350c7c82 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:28:01.509495 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5dc055-d302-440e-bf98-77e9350c7c82 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:28:01.514104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c15379-e157-4a1c-b748-c163bd26d10a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:28:01.514688 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c15379-e157-4a1c-b748-c163bd26d10a admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:28:01.514959 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c15379-e157-4a1c-b748-c163bd26d10a admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:28:01.515228 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c15379-e157-4a1c-b748-c163bd26d10a admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:28:01.515846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c15379-e157-4a1c-b748-c163bd26d10a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:28:01.517750 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c15379-e157-4a1c-b748-c163bd26d10a admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:28:01.545405 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1358/2704] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:28:01 2022] POST /identity/v3/projects => generated 399 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:28:01.549733 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c6b85d-0979-423c-bbd1-e8073a1910ec admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:28:01.560075 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c6b85d-0979-423c-bbd1-e8073a1910ec admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:28:01.615376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a17c2e-f9ac-4c75-9929-88531cef537d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:28:01.615976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a17c2e-f9ac-4c75-9929-88531cef537d admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:28:01.616257 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a17c2e-f9ac-4c75-9929-88531cef537d admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:28:01.616529 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a17c2e-f9ac-4c75-9929-88531cef537d admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:28:01.617156 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a17c2e-f9ac-4c75-9929-88531cef537d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:28:01.619069 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a17c2e-f9ac-4c75-9929-88531cef537d admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:28:01.628071 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:28:01.641385 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1347/2705] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:28:01 2022] POST /identity/v3/users => generated 300 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:28:01.645611 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c15379-e157-4a1c-b748-c163bd26d10a admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:28:01.655561 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c15379-e157-4a1c-b748-c163bd26d10a admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:28:01.659984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f62fc6-e25f-44d4-9cb6-fed6aa9bafa3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:28:01.660565 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f62fc6-e25f-44d4-9cb6-fed6aa9bafa3 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:28:01.660837 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f62fc6-e25f-44d4-9cb6-fed6aa9bafa3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:28:01.661106 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f62fc6-e25f-44d4-9cb6-fed6aa9bafa3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:28:01.661731 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f62fc6-e25f-44d4-9cb6-fed6aa9bafa3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:28:01.663307 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f62fc6-e25f-44d4-9cb6-fed6aa9bafa3 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:28:01.673116 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1359/2706] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:28:01 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:28:01.677880 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a17c2e-f9ac-4c75-9929-88531cef537d admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:28:01.687600 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a17c2e-f9ac-4c75-9929-88531cef537d admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:28:01.692174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74470e06-8161-4b5a-8381-6d1a2379f13c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:28:01.693278 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74470e06-8161-4b5a-8381-6d1a2379f13c admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:28:01.693557 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74470e06-8161-4b5a-8381-6d1a2379f13c admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:28:01.693855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74470e06-8161-4b5a-8381-6d1a2379f13c admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:28:01.694493 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74470e06-8161-4b5a-8381-6d1a2379f13c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:28:01.696052 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74470e06-8161-4b5a-8381-6d1a2379f13c admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:28:01.705767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1348/2707] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:28:01 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:28:01.710288 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f62fc6-e25f-44d4-9cb6-fed6aa9bafa3 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:28:01.719959 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f62fc6-e25f-44d4-9cb6-fed6aa9bafa3 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:28:01.724492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39658254-c031-406b-9056-c900625bbe66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:28:01.725120 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39658254-c031-406b-9056-c900625bbe66 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:28:01.725395 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39658254-c031-406b-9056-c900625bbe66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:28:01.725666 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39658254-c031-406b-9056-c900625bbe66 admin admin] PATH_INFO: `/v3/projects/1d545844780f4499b411252f186139e7/users/0531f67c038046cc96b9f18be22a86b5/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:28:01.740444 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39658254-c031-406b-9056-c900625bbe66 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1d545844780f4499b411252f186139e7, user_id=0531f67c038046cc96b9f18be22a86b5, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:28:01.743062 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39658254-c031-406b-9056-c900625bbe66 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:28:01.745200 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:28:01.758142 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1360/2708] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:28:01 2022] PUT /identity/v3/projects/1d545844780f4499b411252f186139e7/users/0531f67c038046cc96b9f18be22a86b5/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:28:01.769491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118280e0-d475-4748-bbc1-e062091f4662 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:28:01.769813 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118280e0-d475-4748-bbc1-e062091f4662 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:28:01.770104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118280e0-d475-4748-bbc1-e062091f4662 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:28:01.798868 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-118280e0-d475-4748-bbc1-e062091f4662 None None] MFA Rules not processed for user `0531f67c038046cc96b9f18be22a86b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:28:01.800075 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-118280e0-d475-4748-bbc1-e062091f4662 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:28:01.862855 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1349/2709] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:28:01 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:28:01.874628 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39658254-c031-406b-9056-c900625bbe66 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:28:01.876252 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-39658254-c031-406b-9056-c900625bbe66 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:28:01.932488 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39658254-c031-406b-9056-c900625bbe66 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:28:01.936090 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bdafe7-7567-4a4d-86ab-df90a466e061 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:28:01.936680 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bdafe7-7567-4a4d-86ab-df90a466e061 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:28:01.936956 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bdafe7-7567-4a4d-86ab-df90a466e061 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:28:01.937251 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bdafe7-7567-4a4d-86ab-df90a466e061 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:28:01.945804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9bdafe7-7567-4a4d-86ab-df90a466e061 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:28:01.947204 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9bdafe7-7567-4a4d-86ab-df90a466e061 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:28:01.955592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1361/2710] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:28:01 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:28:09.006079 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118280e0-d475-4748-bbc1-e062091f4662 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:28:09.065512 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118280e0-d475-4748-bbc1-e062091f4662 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:28:09.070528 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e83f5b37-73e2-4ca1-bad7-1bcb8b0604ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:28:09.071138 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83f5b37-73e2-4ca1-bad7-1bcb8b0604ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:28:09.071414 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83f5b37-73e2-4ca1-bad7-1bcb8b0604ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:28:09.071702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83f5b37-73e2-4ca1-bad7-1bcb8b0604ac admin admin] PATH_INFO: `/v3/users/0531f67c038046cc96b9f18be22a86b5` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:28:09.072789 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e83f5b37-73e2-4ca1-bad7-1bcb8b0604ac admin admin] RBAC: Authorizing `identity:delete_user(user_id=0531f67c038046cc96b9f18be22a86b5)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:28:09.074705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e83f5b37-73e2-4ca1-bad7-1bcb8b0604ac admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:28:09.121037 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-e83f5b37-73e2-4ca1-bad7-1bcb8b0604ac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0531f67c038046cc96b9f18be22a86b5'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:28:09.127785 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-e83f5b37-73e2-4ca1-bad7-1bcb8b0604ac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0531f67c038046cc96b9f18be22a86b5'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:28:09.128439 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-e83f5b37-73e2-4ca1-bad7-1bcb8b0604ac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0531f67c038046cc96b9f18be22a86b5'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:28:09.136379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-e83f5b37-73e2-4ca1-bad7-1bcb8b0604ac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0531f67c038046cc96b9f18be22a86b5'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:28:09.143308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:INSERT=3,keystone:SELECT=88,keystone:DELETE=9 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:28:09.149857 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-e83f5b37-73e2-4ca1-bad7-1bcb8b0604ac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0531f67c038046cc96b9f18be22a86b5'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:28:09.156899 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1350/2711] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:28:09 2022] DELETE /identity/v3/users/0531f67c038046cc96b9f18be22a86b5 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:28:09.161707 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d7ea8d-2a96-4070-9792-86379749ceb3 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:28:09.162234 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d7ea8d-2a96-4070-9792-86379749ceb3 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:28:09.162341 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d7ea8d-2a96-4070-9792-86379749ceb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:28:09.182074 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-d6d7ea8d-2a96-4070-9792-86379749ceb3 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:28:09.183291 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d6d7ea8d-2a96-4070-9792-86379749ceb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:28:09.242998 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1362/2712] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:28:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:28:09.252814 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e83f5b37-73e2-4ca1-bad7-1bcb8b0604ac admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:28:09.254466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-e83f5b37-73e2-4ca1-bad7-1bcb8b0604ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:28:09.311680 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e83f5b37-73e2-4ca1-bad7-1bcb8b0604ac admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:28:09.315405 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bc2803-0d3f-4f56-911b-68ab62006564 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:28:09.316005 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bc2803-0d3f-4f56-911b-68ab62006564 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:28:09.316279 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bc2803-0d3f-4f56-911b-68ab62006564 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:28:09.316548 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bc2803-0d3f-4f56-911b-68ab62006564 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:28:09.326276 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7bc2803-0d3f-4f56-911b-68ab62006564 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:28:09.327747 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7bc2803-0d3f-4f56-911b-68ab62006564 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:28:09.337184 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1351/2713] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:28:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:28:09.620001 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d7ea8d-2a96-4070-9792-86379749ceb3 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:28:09.622973 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d6d7ea8d-2a96-4070-9792-86379749ceb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:28:09.645119 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d7ea8d-2a96-4070-9792-86379749ceb3 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:28:09.662068 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb073c6-95d9-4095-afd4-2c1a6c464a9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:28:09.662825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb073c6-95d9-4095-afd4-2c1a6c464a9e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:28:09.663200 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb073c6-95d9-4095-afd4-2c1a6c464a9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:28:09.663576 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb073c6-95d9-4095-afd4-2c1a6c464a9e admin admin] PATH_INFO: `/v3/projects/1d545844780f4499b411252f186139e7` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:28:09.665184 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb073c6-95d9-4095-afd4-2c1a6c464a9e admin admin] RBAC: Authorizing `identity:delete_project(project_id=1d545844780f4499b411252f186139e7)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:28:09.667482 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb073c6-95d9-4095-afd4-2c1a6c464a9e admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:28:09.719144 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8cb073c6-95d9-4095-afd4-2c1a6c464a9e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1d545844780f4499b411252f186139e7', 'request_id': 'req-8cb073c6-95d9-4095-afd4-2c1a6c464a9e'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:28:09.725731 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8cb073c6-95d9-4095-afd4-2c1a6c464a9e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1d545844780f4499b411252f186139e7', 'request_id': 'req-8cb073c6-95d9-4095-afd4-2c1a6c464a9e'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:28:09.732236 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8cb073c6-95d9-4095-afd4-2c1a6c464a9e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1d545844780f4499b411252f186139e7', 'request_id': 'req-8cb073c6-95d9-4095-afd4-2c1a6c464a9e'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:28:09.745618 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8cb073c6-95d9-4095-afd4-2c1a6c464a9e admin admin] The token cache is being invalidate because project 1d545844780f4499b411252f186139e7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:28:09.745879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8cb073c6-95d9-4095-afd4-2c1a6c464a9e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:28:09.746489 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-8cb073c6-95d9-4095-afd4-2c1a6c464a9e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:28:09.747954 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1363/2714] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:28:09 2022] DELETE /identity/v3/projects/1d545844780f4499b411252f186139e7 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:28:11.471043 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bc2803-0d3f-4f56-911b-68ab62006564 service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:28:11.472466 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-b7bc2803-0d3f-4f56-911b-68ab62006564 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:28:11.560437 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bc2803-0d3f-4f56-911b-68ab62006564 service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:28:11.617552 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd8cdee-33e2-40da-a42d-6fdb496b42e2 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3a1bc198dfc0486e9e56d4fb28cda05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:28:11.618186 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd8cdee-33e2-40da-a42d-6fdb496b42e2 service placement] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:28:11.618465 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd8cdee-33e2-40da-a42d-6fdb496b42e2 service placement] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:28:11.618737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd8cdee-33e2-40da-a42d-6fdb496b42e2 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:28:11.620457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0bd8cdee-33e2-40da-a42d-6fdb496b42e2 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:28:11.632108 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd8cdee-33e2-40da-a42d-6fdb496b42e2 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:28:11.633656 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd8cdee-33e2-40da-a42d-6fdb496b42e2 service placement] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:28:11.644374 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1352/2715] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:28:11 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:28:17.179777 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7352398f-f28b-4210-87fd-fa6904aca466 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:28:17.180314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7352398f-f28b-4210-87fd-fa6904aca466 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:28:17.180314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7352398f-f28b-4210-87fd-fa6904aca466 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:28:17.198968 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-7352398f-f28b-4210-87fd-fa6904aca466 None None] MFA Rules not processed for user `dfb35bfba72546f3853b9dd9ef7f5078`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:28:17.200167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-7352398f-f28b-4210-87fd-fa6904aca466 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:28:17.212303 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:INSERT=2,keystone:SELECT=91,keystone:DELETE=7 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:28:17.254422 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1364/2716] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:28:17 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:28:21.639383 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:DELETE=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:28:27.247951 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=16 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:29:21.970779 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df38475b-8107-466f-bdea-336fd153ca91 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:21.971375 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df38475b-8107-466f-bdea-336fd153ca91 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:21.971375 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df38475b-8107-466f-bdea-336fd153ca91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:21.992225 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-df38475b-8107-466f-bdea-336fd153ca91 None None] MFA Rules not processed for user `dfb35bfba72546f3853b9dd9ef7f5078`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:29:21.993579 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-df38475b-8107-466f-bdea-336fd153ca91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:29:22.000868 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1353/2717] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:29:21 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:29:22.454337 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7352398f-f28b-4210-87fd-fa6904aca466 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:29:22.459502 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-7352398f-f28b-4210-87fd-fa6904aca466 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:29:22.559257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7352398f-f28b-4210-87fd-fa6904aca466 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:29:22.564503 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0af465-f120-4812-9802-43a721ef3677 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:29:22.565432 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0af465-f120-4812-9802-43a721ef3677 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:22.565712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0af465-f120-4812-9802-43a721ef3677 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:22.566072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0af465-f120-4812-9802-43a721ef3677 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:22.567958 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-3f0af465-f120-4812-9802-43a721ef3677 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:29:22.578875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0af465-f120-4812-9802-43a721ef3677 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:29:22.580319 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0af465-f120-4812-9802-43a721ef3677 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:29:22.595710 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1365/2718] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:29:22 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:29:26.733673 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df38475b-8107-466f-bdea-336fd153ca91 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:29:26.735133 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-df38475b-8107-466f-bdea-336fd153ca91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:29:26.802273 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df38475b-8107-466f-bdea-336fd153ca91 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:29:26.806772 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2054e006-a42a-44ae-bee2-f0dac226c11a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:29:26.807390 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2054e006-a42a-44ae-bee2-f0dac226c11a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:26.807673 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2054e006-a42a-44ae-bee2-f0dac226c11a admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:26.807944 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2054e006-a42a-44ae-bee2-f0dac226c11a admin admin] PATH_INFO: `/v3/users/5ebc33b6c372434287b632d70f0a9e83` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:26.809042 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2054e006-a42a-44ae-bee2-f0dac226c11a admin admin] RBAC: Authorizing `identity:delete_user(user_id=5ebc33b6c372434287b632d70f0a9e83)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:29:26.810948 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2054e006-a42a-44ae-bee2-f0dac226c11a admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:29:26.854576 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-2054e006-a42a-44ae-bee2-f0dac226c11a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5ebc33b6c372434287b632d70f0a9e83'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:29:26.861383 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-2054e006-a42a-44ae-bee2-f0dac226c11a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5ebc33b6c372434287b632d70f0a9e83'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:29:26.862070 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-2054e006-a42a-44ae-bee2-f0dac226c11a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5ebc33b6c372434287b632d70f0a9e83'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:29:26.869841 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-2054e006-a42a-44ae-bee2-f0dac226c11a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5ebc33b6c372434287b632d70f0a9e83'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:29:26.881813 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-2054e006-a42a-44ae-bee2-f0dac226c11a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5ebc33b6c372434287b632d70f0a9e83'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:29:26.887579 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1354/2719] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:29:26 2022] DELETE /identity/v3/users/5ebc33b6c372434287b632d70f0a9e83 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:29:27.060288 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0af465-f120-4812-9802-43a721ef3677 service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:29:27.072110 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0af465-f120-4812-9802-43a721ef3677 service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:29:27.127504 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abf092c-33f7-4ee3-a9e5-08d3ce1ef8d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:29:27.128123 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abf092c-33f7-4ee3-a9e5-08d3ce1ef8d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:27.128396 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abf092c-33f7-4ee3-a9e5-08d3ce1ef8d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:27.128667 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abf092c-33f7-4ee3-a9e5-08d3ce1ef8d7 admin admin] PATH_INFO: `/v3/projects/1e595829629a4dd4a7fccbdabc896cb0` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:27.129781 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abf092c-33f7-4ee3-a9e5-08d3ce1ef8d7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1e595829629a4dd4a7fccbdabc896cb0)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:29:27.131645 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abf092c-33f7-4ee3-a9e5-08d3ce1ef8d7 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:29:27.176928 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-9abf092c-33f7-4ee3-a9e5-08d3ce1ef8d7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1e595829629a4dd4a7fccbdabc896cb0', 'request_id': 'req-9abf092c-33f7-4ee3-a9e5-08d3ce1ef8d7'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:29:27.182383 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-9abf092c-33f7-4ee3-a9e5-08d3ce1ef8d7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1e595829629a4dd4a7fccbdabc896cb0', 'request_id': 'req-9abf092c-33f7-4ee3-a9e5-08d3ce1ef8d7'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:29:27.188692 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-9abf092c-33f7-4ee3-a9e5-08d3ce1ef8d7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1e595829629a4dd4a7fccbdabc896cb0', 'request_id': 'req-9abf092c-33f7-4ee3-a9e5-08d3ce1ef8d7'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:29:27.201486 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-9abf092c-33f7-4ee3-a9e5-08d3ce1ef8d7 admin admin] The token cache is being invalidate because project 1e595829629a4dd4a7fccbdabc896cb0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:29:27.201804 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-9abf092c-33f7-4ee3-a9e5-08d3ce1ef8d7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:29:27.202431 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-9abf092c-33f7-4ee3-a9e5-08d3ce1ef8d7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:29:27.203883 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1366/2720] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:29:27 2022] DELETE /identity/v3/projects/1e595829629a4dd4a7fccbdabc896cb0 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:29:27.208412 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2054e006-a42a-44ae-bee2-f0dac226c11a admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:29:27.209811 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-2054e006-a42a-44ae-bee2-f0dac226c11a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:29:27.222533 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2054e006-a42a-44ae-bee2-f0dac226c11a admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:29:27.276600 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d306051b-0793-4a9a-9cbb-3ddfe94de0d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:29:27.277138 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d306051b-0793-4a9a-9cbb-3ddfe94de0d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:27.277413 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d306051b-0793-4a9a-9cbb-3ddfe94de0d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:27.277680 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d306051b-0793-4a9a-9cbb-3ddfe94de0d1 admin admin] PATH_INFO: `/v3/users/0beee66fb7a945d3a629b596634dd368` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:27.278800 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d306051b-0793-4a9a-9cbb-3ddfe94de0d1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0beee66fb7a945d3a629b596634dd368)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:29:27.280668 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d306051b-0793-4a9a-9cbb-3ddfe94de0d1 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:29:27.294251 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:29:27.325930 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-d306051b-0793-4a9a-9cbb-3ddfe94de0d1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0beee66fb7a945d3a629b596634dd368'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:29:27.332655 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-d306051b-0793-4a9a-9cbb-3ddfe94de0d1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0beee66fb7a945d3a629b596634dd368'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:29:27.333873 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-d306051b-0793-4a9a-9cbb-3ddfe94de0d1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0beee66fb7a945d3a629b596634dd368'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:29:27.341578 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-d306051b-0793-4a9a-9cbb-3ddfe94de0d1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0beee66fb7a945d3a629b596634dd368'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:29:27.354195 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-d306051b-0793-4a9a-9cbb-3ddfe94de0d1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0beee66fb7a945d3a629b596634dd368'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:29:27.360011 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1355/2721] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:29:27 2022] DELETE /identity/v3/users/0beee66fb7a945d3a629b596634dd368 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:29:27.545560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abf092c-33f7-4ee3-a9e5-08d3ce1ef8d7 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:29:27.547587 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-9abf092c-33f7-4ee3-a9e5-08d3ce1ef8d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:29:27.566345 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abf092c-33f7-4ee3-a9e5-08d3ce1ef8d7 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:29:27.579746 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:29:27.631378 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf6a4a9-031b-4128-9161-82133569e113 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:29:27.632010 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf6a4a9-031b-4128-9161-82133569e113 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:27.632286 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf6a4a9-031b-4128-9161-82133569e113 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:27.632558 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf6a4a9-031b-4128-9161-82133569e113 admin admin] PATH_INFO: `/v3/projects/4af948c97bfd43789719fb833d38e164` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:27.633661 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf6a4a9-031b-4128-9161-82133569e113 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4af948c97bfd43789719fb833d38e164)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:29:27.635567 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf6a4a9-031b-4128-9161-82133569e113 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:29:27.681198 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-caf6a4a9-031b-4128-9161-82133569e113 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4af948c97bfd43789719fb833d38e164', 'request_id': 'req-caf6a4a9-031b-4128-9161-82133569e113'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:29:27.686711 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-caf6a4a9-031b-4128-9161-82133569e113 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4af948c97bfd43789719fb833d38e164', 'request_id': 'req-caf6a4a9-031b-4128-9161-82133569e113'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:29:27.693413 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-caf6a4a9-031b-4128-9161-82133569e113 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4af948c97bfd43789719fb833d38e164', 'request_id': 'req-caf6a4a9-031b-4128-9161-82133569e113'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:29:27.707277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-caf6a4a9-031b-4128-9161-82133569e113 admin admin] The token cache is being invalidate because project 4af948c97bfd43789719fb833d38e164 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:29:27.707564 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-caf6a4a9-031b-4128-9161-82133569e113 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:29:27.708238 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-caf6a4a9-031b-4128-9161-82133569e113 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:29:27.709814 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1367/2722] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:29:27 2022] DELETE /identity/v3/projects/4af948c97bfd43789719fb833d38e164 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:29:27.714938 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f46ea52-d456-43ef-a81b-8d5704e489f4 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:27.715244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f46ea52-d456-43ef-a81b-8d5704e489f4 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:27.715513 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f46ea52-d456-43ef-a81b-8d5704e489f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:27.736413 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-3f46ea52-d456-43ef-a81b-8d5704e489f4 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:29:27.737653 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-3f46ea52-d456-43ef-a81b-8d5704e489f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:29:27.796977 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1356/2723] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:29:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:29:27.805267 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1f8ada-2ab5-458d-916f-fd5dde9f39b1 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:27.805550 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1f8ada-2ab5-458d-916f-fd5dde9f39b1 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:27.805842 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1f8ada-2ab5-458d-916f-fd5dde9f39b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:27.829281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-2a1f8ada-2ab5-458d-916f-fd5dde9f39b1 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:29:27.830539 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-2a1f8ada-2ab5-458d-916f-fd5dde9f39b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:29:27.839400 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1368/2724] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:29:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:29:27.843370 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f46ea52-d456-43ef-a81b-8d5704e489f4 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:29:27.853964 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f46ea52-d456-43ef-a81b-8d5704e489f4 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:29:27.861622 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ab0ac6-370b-4ce4-a70c-61bbff7b2547 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:29:27.869129 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ab0ac6-370b-4ce4-a70c-61bbff7b2547 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:27.869666 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ab0ac6-370b-4ce4-a70c-61bbff7b2547 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:27.874149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ab0ac6-370b-4ce4-a70c-61bbff7b2547 admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:27.874927 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ab0ac6-370b-4ce4-a70c-61bbff7b2547 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:29:27.875306 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ab0ac6-370b-4ce4-a70c-61bbff7b2547 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:29:27.877234 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ab0ac6-370b-4ce4-a70c-61bbff7b2547 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:29:27.895980 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1357/2725] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:29:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:29:27.897065 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1f8ada-2ab5-458d-916f-fd5dde9f39b1 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:29:27.906876 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1f8ada-2ab5-458d-916f-fd5dde9f39b1 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:29:27.911610 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aadf8d2-2146-464a-8bf0-82f0024294e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:29:27.912274 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aadf8d2-2146-464a-8bf0-82f0024294e1 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:27.912612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aadf8d2-2146-464a-8bf0-82f0024294e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:27.912941 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aadf8d2-2146-464a-8bf0-82f0024294e1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:27.913621 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aadf8d2-2146-464a-8bf0-82f0024294e1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:29:27.915810 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aadf8d2-2146-464a-8bf0-82f0024294e1 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:29:27.943796 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1369/2726] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:29:27 2022] POST /identity/v3/projects => generated 375 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:29:27.948089 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ab0ac6-370b-4ce4-a70c-61bbff7b2547 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:29:27.959313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ab0ac6-370b-4ce4-a70c-61bbff7b2547 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:29:28.020700 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c4a0b9-f954-4e31-bfd5-8d08267dc5b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:29:28.021496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c4a0b9-f954-4e31-bfd5-8d08267dc5b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:28.021858 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c4a0b9-f954-4e31-bfd5-8d08267dc5b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:28.022194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c4a0b9-f954-4e31-bfd5-8d08267dc5b2 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:28.022924 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c4a0b9-f954-4e31-bfd5-8d08267dc5b2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:29:28.024854 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c4a0b9-f954-4e31-bfd5-8d08267dc5b2 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:29:28.046256 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1358/2727] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:29:27 2022] POST /identity/v3/users => generated 288 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:29:28.050465 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aadf8d2-2146-464a-8bf0-82f0024294e1 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:29:28.060907 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aadf8d2-2146-464a-8bf0-82f0024294e1 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:29:28.065433 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91095f18-fb93-4aca-8324-b21ade9c66c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:29:28.066041 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91095f18-fb93-4aca-8324-b21ade9c66c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:28.066314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91095f18-fb93-4aca-8324-b21ade9c66c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:28.066581 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91095f18-fb93-4aca-8324-b21ade9c66c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:28.067785 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91095f18-fb93-4aca-8324-b21ade9c66c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:29:28.069351 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91095f18-fb93-4aca-8324-b21ade9c66c0 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:29:28.079351 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1370/2728] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:29:28 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:29:28.083107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c4a0b9-f954-4e31-bfd5-8d08267dc5b2 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:29:28.092717 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c4a0b9-f954-4e31-bfd5-8d08267dc5b2 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:29:28.097151 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37fced5c-7417-48ac-bfcb-d7fa3214499d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:29:28.097729 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fced5c-7417-48ac-bfcb-d7fa3214499d admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:28.098034 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fced5c-7417-48ac-bfcb-d7fa3214499d admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:28.098301 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fced5c-7417-48ac-bfcb-d7fa3214499d admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:28.098959 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37fced5c-7417-48ac-bfcb-d7fa3214499d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:29:28.100528 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37fced5c-7417-48ac-bfcb-d7fa3214499d admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:29:28.102519 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:29:28.112469 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1359/2729] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:29:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:29:28.116946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91095f18-fb93-4aca-8324-b21ade9c66c0 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:29:28.126855 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91095f18-fb93-4aca-8324-b21ade9c66c0 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:29:28.131976 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49f9322-348e-463c-bb68-26f2357e8bff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:29:28.132711 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49f9322-348e-463c-bb68-26f2357e8bff admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:28.133011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49f9322-348e-463c-bb68-26f2357e8bff admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:28.133329 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49f9322-348e-463c-bb68-26f2357e8bff admin admin] PATH_INFO: `/v3/projects/53d86b6e8a8343fca03ca6d38200f23f/users/44a710d801334b20b9f433eaad61cbc6/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:28.147696 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49f9322-348e-463c-bb68-26f2357e8bff admin admin] RBAC: Authorizing `identity:create_grant(project_id=53d86b6e8a8343fca03ca6d38200f23f, user_id=44a710d801334b20b9f433eaad61cbc6, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:29:28.150321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49f9322-348e-463c-bb68-26f2357e8bff admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:29:28.162599 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1371/2730] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:29:28 2022] PUT /identity/v3/projects/53d86b6e8a8343fca03ca6d38200f23f/users/44a710d801334b20b9f433eaad61cbc6/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:29:28.167503 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b3a619-9348-4ea3-a309-96383dfacc1d None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:28.167787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b3a619-9348-4ea3-a309-96383dfacc1d None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:28.168061 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b3a619-9348-4ea3-a309-96383dfacc1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:28.187303 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-15b3a619-9348-4ea3-a309-96383dfacc1d None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:29:28.188480 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-15b3a619-9348-4ea3-a309-96383dfacc1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:29:28.248706 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1360/2731] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:29:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:29:28.258298 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49f9322-348e-463c-bb68-26f2357e8bff admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:29:28.259980 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-c49f9322-348e-463c-bb68-26f2357e8bff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:29:28.304072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:29:28.320206 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49f9322-348e-463c-bb68-26f2357e8bff admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:29:28.323876 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a47482-feaf-435c-a741-430b421e04ea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:29:28.324477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a47482-feaf-435c-a741-430b421e04ea service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:28.324755 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a47482-feaf-435c-a741-430b421e04ea service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:28.325028 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a47482-feaf-435c-a741-430b421e04ea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:28.334396 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a47482-feaf-435c-a741-430b421e04ea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:29:28.335826 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a47482-feaf-435c-a741-430b421e04ea service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:29:28.345295 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1372/2732] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:29:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:29:32.546936 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb549e6-9fc0-4160-a76c-588cca91318a None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:32.547450 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb549e6-9fc0-4160-a76c-588cca91318a None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:32.547567 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb549e6-9fc0-4160-a76c-588cca91318a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:32.577241 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-1cb549e6-9fc0-4160-a76c-588cca91318a None None] MFA Rules not processed for user `44a710d801334b20b9f433eaad61cbc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:29:32.578504 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-1cb549e6-9fc0-4160-a76c-588cca91318a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:29:32.668115 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1361/2733] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:29:32 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:29:32.670033 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a47482-feaf-435c-a741-430b421e04ea service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:29:32.683099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a47482-feaf-435c-a741-430b421e04ea service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:29:32.688997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c010146-557b-471a-b5f3-2438d06df036 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:29:32.689818 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c010146-557b-471a-b5f3-2438d06df036 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:32.690345 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c010146-557b-471a-b5f3-2438d06df036 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:32.690849 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c010146-557b-471a-b5f3-2438d06df036 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:32.691886 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c010146-557b-471a-b5f3-2438d06df036 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:29:32.694775 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c010146-557b-471a-b5f3-2438d06df036 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:29:32.727718 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1373/2734] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:29:32 2022] POST /identity/v3/projects => generated 379 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:29:32.731934 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb549e6-9fc0-4160-a76c-588cca91318a None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:29:32.744287 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb549e6-9fc0-4160-a76c-588cca91318a None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:29:32.803094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff003fb-2a07-4e3a-9a29-fee03ea69ee2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:29:32.803826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff003fb-2a07-4e3a-9a29-fee03ea69ee2 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:32.804206 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff003fb-2a07-4e3a-9a29-fee03ea69ee2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:32.804688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff003fb-2a07-4e3a-9a29-fee03ea69ee2 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:32.805531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff003fb-2a07-4e3a-9a29-fee03ea69ee2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:29:32.807694 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff003fb-2a07-4e3a-9a29-fee03ea69ee2 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:29:32.838658 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1362/2735] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:29:32 2022] POST /identity/v3/users => generated 296 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:29:32.840293 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c010146-557b-471a-b5f3-2438d06df036 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:29:32.850907 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c010146-557b-471a-b5f3-2438d06df036 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:29:32.856189 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b002e57-bf96-43c4-a1df-3d78db0bd0d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:29:32.856883 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b002e57-bf96-43c4-a1df-3d78db0bd0d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:32.857249 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b002e57-bf96-43c4-a1df-3d78db0bd0d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:32.857588 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b002e57-bf96-43c4-a1df-3d78db0bd0d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:32.858288 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b002e57-bf96-43c4-a1df-3d78db0bd0d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:29:32.859889 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b002e57-bf96-43c4-a1df-3d78db0bd0d4 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:29:32.873807 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1374/2736] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:29:32 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:29:32.877987 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff003fb-2a07-4e3a-9a29-fee03ea69ee2 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:29:32.889664 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff003fb-2a07-4e3a-9a29-fee03ea69ee2 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:29:32.894849 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b70245-4018-4f27-899f-a7a42baecef4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:29:32.899436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b70245-4018-4f27-899f-a7a42baecef4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:32.899436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b70245-4018-4f27-899f-a7a42baecef4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:32.899436 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b70245-4018-4f27-899f-a7a42baecef4 admin admin] PATH_INFO: `/v3/projects/8069df7a2e984db0ac8925e1f0c242d8/users/e218879cb193406c9339b38dd833d929/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:32.923579 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b70245-4018-4f27-899f-a7a42baecef4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8069df7a2e984db0ac8925e1f0c242d8, user_id=e218879cb193406c9339b38dd833d929, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:29:32.926302 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b70245-4018-4f27-899f-a7a42baecef4 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:29:32.928799 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:29:32.945408 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1363/2737] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:29:32 2022] PUT /identity/v3/projects/8069df7a2e984db0ac8925e1f0c242d8/users/e218879cb193406c9339b38dd833d929/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:29:32.949952 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b002e57-bf96-43c4-a1df-3d78db0bd0d4 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:29:32.964028 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b002e57-bf96-43c4-a1df-3d78db0bd0d4 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:29:33.037111 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfe957b-339a-4330-b5dd-2232d1331111 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:29:33.037771 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfe957b-339a-4330-b5dd-2232d1331111 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:33.038046 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfe957b-339a-4330-b5dd-2232d1331111 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:33.038324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfe957b-339a-4330-b5dd-2232d1331111 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:33.038971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbfe957b-339a-4330-b5dd-2232d1331111 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:29:33.040599 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbfe957b-339a-4330-b5dd-2232d1331111 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:29:33.052016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1375/2738] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:29:32 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:29:33.055862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b70245-4018-4f27-899f-a7a42baecef4 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:29:33.069765 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b70245-4018-4f27-899f-a7a42baecef4 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:29:33.075120 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2c7297-f9f5-41eb-8bb6-48a70332bec9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:29:33.076016 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2c7297-f9f5-41eb-8bb6-48a70332bec9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:33.076380 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2c7297-f9f5-41eb-8bb6-48a70332bec9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:33.076725 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2c7297-f9f5-41eb-8bb6-48a70332bec9 admin admin] PATH_INFO: `/v3/projects/8069df7a2e984db0ac8925e1f0c242d8/users/e218879cb193406c9339b38dd833d929/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:33.078893 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2c7297-f9f5-41eb-8bb6-48a70332bec9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8069df7a2e984db0ac8925e1f0c242d8, user_id=e218879cb193406c9339b38dd833d929, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:29:33.081496 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2c7297-f9f5-41eb-8bb6-48a70332bec9 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:29:33.098367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1364/2739] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:29:33 2022] PUT /identity/v3/projects/8069df7a2e984db0ac8925e1f0c242d8/users/e218879cb193406c9339b38dd833d929/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:29:38.352451 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa84b4e-f4ee-435c-9193-99ac1af1823e None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:38.353018 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa84b4e-f4ee-435c-9193-99ac1af1823e None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:38.353182 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa84b4e-f4ee-435c-9193-99ac1af1823e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:38.390264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-3aa84b4e-f4ee-435c-9193-99ac1af1823e None None] MFA Rules not processed for user `e218879cb193406c9339b38dd833d929`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:29:38.391398 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-3aa84b4e-f4ee-435c-9193-99ac1af1823e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:29:38.465421 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1376/2740] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:29:38 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:29:38.480721 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2c7297-f9f5-41eb-8bb6-48a70332bec9 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:29:38.482807 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-2f2c7297-f9f5-41eb-8bb6-48a70332bec9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:29:38.588705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2c7297-f9f5-41eb-8bb6-48a70332bec9 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:29:38.594152 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d40843-1cfe-476d-a8d5-721cf14a3ecb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:29:38.594749 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d40843-1cfe-476d-a8d5-721cf14a3ecb service nova] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:38.595023 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d40843-1cfe-476d-a8d5-721cf14a3ecb service nova] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:38.595289 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d40843-1cfe-476d-a8d5-721cf14a3ecb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:38.654104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d40843-1cfe-476d-a8d5-721cf14a3ecb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:29:38.655680 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d40843-1cfe-476d-a8d5-721cf14a3ecb service nova] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:29:38.667098 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1365/2741] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:29:38 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:29:48.463585 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=85,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:29:48.662040 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:INSERT=2,keystone:SELECT=75 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:29:55.039422 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6b4176-668a-4896-bc81-6f8e9fde3b73 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:55.039940 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6b4176-668a-4896-bc81-6f8e9fde3b73 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:55.039940 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6b4176-668a-4896-bc81-6f8e9fde3b73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:55.059563 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-2e6b4176-668a-4896-bc81-6f8e9fde3b73 None None] MFA Rules not processed for user `44a710d801334b20b9f433eaad61cbc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:29:55.060755 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-2e6b4176-668a-4896-bc81-6f8e9fde3b73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:29:55.070117 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1377/2742] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:29:55 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:29:59.278152 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d40843-1cfe-476d-a8d5-721cf14a3ecb service nova] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:29:59.331932 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d40843-1cfe-476d-a8d5-721cf14a3ecb service nova] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:29:59.336135 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35065da8-3c7d-46c7-8b5d-2f31dac4e5a4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:29:59.336738 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35065da8-3c7d-46c7-8b5d-2f31dac4e5a4 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:29:59.337017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35065da8-3c7d-46c7-8b5d-2f31dac4e5a4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:29:59.337287 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35065da8-3c7d-46c7-8b5d-2f31dac4e5a4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:29:59.339002 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-35065da8-3c7d-46c7-8b5d-2f31dac4e5a4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:29:59.352569 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35065da8-3c7d-46c7-8b5d-2f31dac4e5a4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:29:59.354012 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35065da8-3c7d-46c7-8b5d-2f31dac4e5a4 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:29:59.365678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1366/2743] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:29:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:30:02.973210 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6b4176-668a-4896-bc81-6f8e9fde3b73 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:30:02.975181 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-2e6b4176-668a-4896-bc81-6f8e9fde3b73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:30:02.986729 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6b4176-668a-4896-bc81-6f8e9fde3b73 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:30:02.990774 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916be05b-521b-4e43-a2c7-e699d971dc90 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:30:02.991354 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916be05b-521b-4e43-a2c7-e699d971dc90 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:30:02.991624 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916be05b-521b-4e43-a2c7-e699d971dc90 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:30:02.991889 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916be05b-521b-4e43-a2c7-e699d971dc90 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:30:02.993864 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-916be05b-521b-4e43-a2c7-e699d971dc90 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:30:03.006658 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916be05b-521b-4e43-a2c7-e699d971dc90 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:30:03.008052 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916be05b-521b-4e43-a2c7-e699d971dc90 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:30:03.018543 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1378/2744] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:30:02 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:30:09.360218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=26 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:30:11.527840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35065da8-3c7d-46c7-8b5d-2f31dac4e5a4 service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:30:11.536456 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35065da8-3c7d-46c7-8b5d-2f31dac4e5a4 service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:30:11.540008 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24433105-cee5-4a6b-bb20-684df8738b5e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:30:11.540593 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24433105-cee5-4a6b-bb20-684df8738b5e service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:30:11.540870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24433105-cee5-4a6b-bb20-684df8738b5e service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:30:11.541138 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24433105-cee5-4a6b-bb20-684df8738b5e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:30:11.550046 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24433105-cee5-4a6b-bb20-684df8738b5e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:30:11.552380 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24433105-cee5-4a6b-bb20-684df8738b5e service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:30:11.561492 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1367/2745] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:30:11 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:30:13.013853 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=9 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:30:21.557411 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=6 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:30:31.461243 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8baaed-39db-443e-b1db-1be7eb35e6d0 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:30:31.461814 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8baaed-39db-443e-b1db-1be7eb35e6d0 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:30:31.462072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8baaed-39db-443e-b1db-1be7eb35e6d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:30:31.480949 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-fb8baaed-39db-443e-b1db-1be7eb35e6d0 None None] MFA Rules not processed for user `dfb35bfba72546f3853b9dd9ef7f5078`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:30:31.482180 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-fb8baaed-39db-443e-b1db-1be7eb35e6d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:30:31.534757 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1379/2746] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:30:31 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:30:33.930693 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24433105-cee5-4a6b-bb20-684df8738b5e service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:30:33.940097 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24433105-cee5-4a6b-bb20-684df8738b5e service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:30:33.943807 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390d8667-7345-4f0c-8a8e-b4d67cfbb43c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:30:33.944479 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390d8667-7345-4f0c-8a8e-b4d67cfbb43c service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:30:33.944817 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390d8667-7345-4f0c-8a8e-b4d67cfbb43c service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:30:33.945151 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390d8667-7345-4f0c-8a8e-b4d67cfbb43c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:30:33.946992 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-390d8667-7345-4f0c-8a8e-b4d67cfbb43c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:30:33.957688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390d8667-7345-4f0c-8a8e-b4d67cfbb43c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:30:33.959225 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390d8667-7345-4f0c-8a8e-b4d67cfbb43c service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:30:33.967972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1368/2747] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:30:33 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:30:41.527671 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=25 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:30:43.964922 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=6 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:32.285146 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8baaed-39db-443e-b1db-1be7eb35e6d0 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:32.287679 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-fb8baaed-39db-443e-b1db-1be7eb35e6d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:32.314797 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8baaed-39db-443e-b1db-1be7eb35e6d0 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:32.321893 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d61340c-9957-4147-8ccd-0be9bc363567 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:32.322522 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d61340c-9957-4147-8ccd-0be9bc363567 service nova] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:32.322799 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d61340c-9957-4147-8ccd-0be9bc363567 service nova] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:32.323282 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d61340c-9957-4147-8ccd-0be9bc363567 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:32.325240 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-3d61340c-9957-4147-8ccd-0be9bc363567 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:32.341061 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d61340c-9957-4147-8ccd-0be9bc363567 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:32.344359 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d61340c-9957-4147-8ccd-0be9bc363567 service nova] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:32.370149 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1380/2748] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:31:32 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:31:40.760554 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390d8667-7345-4f0c-8a8e-b4d67cfbb43c service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:40.762697 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-390d8667-7345-4f0c-8a8e-b4d67cfbb43c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:40.818067 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d61340c-9957-4147-8ccd-0be9bc363567 service nova] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:40.819996 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-3d61340c-9957-4147-8ccd-0be9bc363567 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:40.840584 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390d8667-7345-4f0c-8a8e-b4d67cfbb43c service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:40.845524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee651184-4473-4a40-a082-a235a0202361 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3a1bc198dfc0486e9e56d4fb28cda05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:40.846199 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee651184-4473-4a40-a082-a235a0202361 service placement] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:40.846494 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee651184-4473-4a40-a082-a235a0202361 service placement] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:40.846746 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee651184-4473-4a40-a082-a235a0202361 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:40.848499 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ee651184-4473-4a40-a082-a235a0202361 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:40.860394 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee651184-4473-4a40-a082-a235a0202361 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:40.862003 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee651184-4473-4a40-a082-a235a0202361 service placement] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:40.874195 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1369/2749] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:31:40 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:31:40.889856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d61340c-9957-4147-8ccd-0be9bc363567 service nova] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:40.894301 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9803036-4ddf-4dbd-b243-76f8463f1797 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:40.894945 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9803036-4ddf-4dbd-b243-76f8463f1797 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:40.895217 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9803036-4ddf-4dbd-b243-76f8463f1797 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:40.895489 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9803036-4ddf-4dbd-b243-76f8463f1797 admin admin] PATH_INFO: `/v3/users/dfb35bfba72546f3853b9dd9ef7f5078` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:40.896591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9803036-4ddf-4dbd-b243-76f8463f1797 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dfb35bfba72546f3853b9dd9ef7f5078)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:40.898573 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9803036-4ddf-4dbd-b243-76f8463f1797 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:40.948863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-d9803036-4ddf-4dbd-b243-76f8463f1797 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dfb35bfba72546f3853b9dd9ef7f5078'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:40.955792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-d9803036-4ddf-4dbd-b243-76f8463f1797 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dfb35bfba72546f3853b9dd9ef7f5078'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:40.956459 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-d9803036-4ddf-4dbd-b243-76f8463f1797 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dfb35bfba72546f3853b9dd9ef7f5078'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:40.963419 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-d9803036-4ddf-4dbd-b243-76f8463f1797 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dfb35bfba72546f3853b9dd9ef7f5078'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:40.975752 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-d9803036-4ddf-4dbd-b243-76f8463f1797 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dfb35bfba72546f3853b9dd9ef7f5078'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:40.981629 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1381/2750] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:31:40 2022] DELETE /identity/v3/users/dfb35bfba72546f3853b9dd9ef7f5078 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:31:41.161979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee651184-4473-4a40-a082-a235a0202361 service placement] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:41.173117 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee651184-4473-4a40-a082-a235a0202361 service placement] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:41.238052 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c4ffd4-e8df-4390-a596-cb238b2dfb5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:41.239220 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c4ffd4-e8df-4390-a596-cb238b2dfb5b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:41.239495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c4ffd4-e8df-4390-a596-cb238b2dfb5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:41.239760 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c4ffd4-e8df-4390-a596-cb238b2dfb5b admin admin] PATH_INFO: `/v3/projects/d7c74ffec54740eca3cd6b707f1cb732` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:41.240973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c4ffd4-e8df-4390-a596-cb238b2dfb5b admin admin] RBAC: Authorizing `identity:delete_project(project_id=d7c74ffec54740eca3cd6b707f1cb732)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:41.242885 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c4ffd4-e8df-4390-a596-cb238b2dfb5b admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:41.310121 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-81c4ffd4-e8df-4390-a596-cb238b2dfb5b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd7c74ffec54740eca3cd6b707f1cb732', 'request_id': 'req-81c4ffd4-e8df-4390-a596-cb238b2dfb5b'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:41.315597 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-81c4ffd4-e8df-4390-a596-cb238b2dfb5b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd7c74ffec54740eca3cd6b707f1cb732', 'request_id': 'req-81c4ffd4-e8df-4390-a596-cb238b2dfb5b'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:41.321835 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-81c4ffd4-e8df-4390-a596-cb238b2dfb5b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd7c74ffec54740eca3cd6b707f1cb732', 'request_id': 'req-81c4ffd4-e8df-4390-a596-cb238b2dfb5b'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:41.334580 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-81c4ffd4-e8df-4390-a596-cb238b2dfb5b admin admin] The token cache is being invalidate because project d7c74ffec54740eca3cd6b707f1cb732 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:31:41.334870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-81c4ffd4-e8df-4390-a596-cb238b2dfb5b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:41.335492 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-81c4ffd4-e8df-4390-a596-cb238b2dfb5b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:41.336965 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1370/2751] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:31:41 2022] DELETE /identity/v3/projects/d7c74ffec54740eca3cd6b707f1cb732 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:31:41.342031 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc5f61d-d4be-4d84-9b4e-c9d0ec8beb15 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:41.342320 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc5f61d-d4be-4d84-9b4e-c9d0ec8beb15 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:41.342597 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc5f61d-d4be-4d84-9b4e-c9d0ec8beb15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:41.362938 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-0fc5f61d-d4be-4d84-9b4e-c9d0ec8beb15 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:31:41.364223 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-0fc5f61d-d4be-4d84-9b4e-c9d0ec8beb15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:41.422939 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1382/2752] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:31:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:31:41.430840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb269e8-877d-417e-a071-9cf378199caa None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:41.431127 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb269e8-877d-417e-a071-9cf378199caa None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:41.431402 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb269e8-877d-417e-a071-9cf378199caa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:41.451525 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-9bb269e8-877d-417e-a071-9cf378199caa None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:31:41.452738 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-9bb269e8-877d-417e-a071-9cf378199caa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:41.461042 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1371/2753] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:31:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:31:41.465373 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc5f61d-d4be-4d84-9b4e-c9d0ec8beb15 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:41.475799 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc5f61d-d4be-4d84-9b4e-c9d0ec8beb15 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:41.480519 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6bea1e-1bff-495a-8e8a-6bf3beaebe4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:41.481107 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6bea1e-1bff-495a-8e8a-6bf3beaebe4a admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:41.481383 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6bea1e-1bff-495a-8e8a-6bf3beaebe4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:41.481652 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6bea1e-1bff-495a-8e8a-6bf3beaebe4a admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:41.482320 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6bea1e-1bff-495a-8e8a-6bf3beaebe4a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:31:41.482621 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6bea1e-1bff-495a-8e8a-6bf3beaebe4a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:41.484186 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6bea1e-1bff-495a-8e8a-6bf3beaebe4a admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:41.485722 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:41.500300 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1383/2754] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:31:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:31:41.501415 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb269e8-877d-417e-a071-9cf378199caa None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:41.510579 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb269e8-877d-417e-a071-9cf378199caa None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:41.515184 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f05921-1307-4445-a640-7ff17331ac27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:41.515768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f05921-1307-4445-a640-7ff17331ac27 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:41.516042 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f05921-1307-4445-a640-7ff17331ac27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:41.516311 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f05921-1307-4445-a640-7ff17331ac27 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:41.516945 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f05921-1307-4445-a640-7ff17331ac27 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:41.518847 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f05921-1307-4445-a640-7ff17331ac27 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:41.530617 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:41.556816 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1372/2755] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:31:41 2022] POST /identity/v3/projects => generated 371 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:31:41.560864 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6bea1e-1bff-495a-8e8a-6bf3beaebe4a admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:41.570863 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6bea1e-1bff-495a-8e8a-6bf3beaebe4a admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:41.626438 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154a9a7d-e6c7-4a8a-be54-91abd6846b22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:41.627037 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154a9a7d-e6c7-4a8a-be54-91abd6846b22 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:41.627314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154a9a7d-e6c7-4a8a-be54-91abd6846b22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:41.627585 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154a9a7d-e6c7-4a8a-be54-91abd6846b22 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:41.628218 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-154a9a7d-e6c7-4a8a-be54-91abd6846b22 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:41.630771 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-154a9a7d-e6c7-4a8a-be54-91abd6846b22 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:41.652589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1384/2756] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:31:41 2022] POST /identity/v3/users => generated 286 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:31:41.657115 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f05921-1307-4445-a640-7ff17331ac27 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:41.667690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f05921-1307-4445-a640-7ff17331ac27 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:41.672310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2ed53b-4a39-4f2a-b01a-128da9ae30dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:41.672904 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2ed53b-4a39-4f2a-b01a-128da9ae30dc admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:41.673185 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2ed53b-4a39-4f2a-b01a-128da9ae30dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:41.673458 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2ed53b-4a39-4f2a-b01a-128da9ae30dc admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:41.674134 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2ed53b-4a39-4f2a-b01a-128da9ae30dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:41.676327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2ed53b-4a39-4f2a-b01a-128da9ae30dc admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:41.686197 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1373/2757] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:31:41 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:31:41.690239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154a9a7d-e6c7-4a8a-be54-91abd6846b22 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:41.700281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154a9a7d-e6c7-4a8a-be54-91abd6846b22 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:41.705093 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0849a080-d7e9-4816-97a8-68beccbb5f5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:41.705712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0849a080-d7e9-4816-97a8-68beccbb5f5c admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:41.706040 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0849a080-d7e9-4816-97a8-68beccbb5f5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:41.706298 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0849a080-d7e9-4816-97a8-68beccbb5f5c admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:41.706939 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0849a080-d7e9-4816-97a8-68beccbb5f5c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:41.708548 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0849a080-d7e9-4816-97a8-68beccbb5f5c admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:41.719272 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1385/2758] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:31:41 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:31:41.724030 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2ed53b-4a39-4f2a-b01a-128da9ae30dc admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:41.734905 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2ed53b-4a39-4f2a-b01a-128da9ae30dc admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:41.740221 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc104eb5-377a-461e-a09f-316757774318 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:41.740885 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc104eb5-377a-461e-a09f-316757774318 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:41.741168 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc104eb5-377a-461e-a09f-316757774318 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:41.741446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc104eb5-377a-461e-a09f-316757774318 admin admin] PATH_INFO: `/v3/projects/b86c8a654e564d8fbe35c070363144f1/users/3bb2b5e770924b7383739a66e9308638/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:41.756190 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc104eb5-377a-461e-a09f-316757774318 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b86c8a654e564d8fbe35c070363144f1, user_id=3bb2b5e770924b7383739a66e9308638, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:41.758980 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc104eb5-377a-461e-a09f-316757774318 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:41.773120 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1374/2759] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:31:41 2022] PUT /identity/v3/projects/b86c8a654e564d8fbe35c070363144f1/users/3bb2b5e770924b7383739a66e9308638/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:31:41.783667 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db650ac-ce62-4a87-b5ab-6766afd00570 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:41.783948 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db650ac-ce62-4a87-b5ab-6766afd00570 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:41.784220 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db650ac-ce62-4a87-b5ab-6766afd00570 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:41.813761 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-7db650ac-ce62-4a87-b5ab-6766afd00570 None None] MFA Rules not processed for user `3bb2b5e770924b7383739a66e9308638`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:31:41.815075 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-7db650ac-ce62-4a87-b5ab-6766afd00570 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:41.879257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1386/2760] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:31:41 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:31:41.890217 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc104eb5-377a-461e-a09f-316757774318 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:41.891968 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-fc104eb5-377a-461e-a09f-316757774318 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:41.966756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc104eb5-377a-461e-a09f-316757774318 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:41.973093 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8ef798-f5a6-4b81-beb6-7529079d6151 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:41.973857 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8ef798-f5a6-4b81-beb6-7529079d6151 service nova] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:41.974142 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8ef798-f5a6-4b81-beb6-7529079d6151 service nova] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:41.974419 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8ef798-f5a6-4b81-beb6-7529079d6151 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:41.983888 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8ef798-f5a6-4b81-beb6-7529079d6151 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:41.985506 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8ef798-f5a6-4b81-beb6-7529079d6151 service nova] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:41.995194 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1375/2761] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:31:41 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:31:44.672821 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db650ac-ce62-4a87-b5ab-6766afd00570 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:44.731263 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db650ac-ce62-4a87-b5ab-6766afd00570 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:44.735728 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db935ab0-f12b-4031-b23c-ce6223ee0cfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:44.736340 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db935ab0-f12b-4031-b23c-ce6223ee0cfb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:44.736617 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db935ab0-f12b-4031-b23c-ce6223ee0cfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:44.736887 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db935ab0-f12b-4031-b23c-ce6223ee0cfb admin admin] PATH_INFO: `/v3/users/3bb2b5e770924b7383739a66e9308638` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:44.738016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db935ab0-f12b-4031-b23c-ce6223ee0cfb admin admin] RBAC: Authorizing `identity:delete_user(user_id=3bb2b5e770924b7383739a66e9308638)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:44.739896 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db935ab0-f12b-4031-b23c-ce6223ee0cfb admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:44.742239 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:44.786977 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-db935ab0-f12b-4031-b23c-ce6223ee0cfb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3bb2b5e770924b7383739a66e9308638'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:44.793728 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-db935ab0-f12b-4031-b23c-ce6223ee0cfb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3bb2b5e770924b7383739a66e9308638'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:44.794401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-db935ab0-f12b-4031-b23c-ce6223ee0cfb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3bb2b5e770924b7383739a66e9308638'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:44.801396 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-db935ab0-f12b-4031-b23c-ce6223ee0cfb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3bb2b5e770924b7383739a66e9308638'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:44.813220 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-db935ab0-f12b-4031-b23c-ce6223ee0cfb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3bb2b5e770924b7383739a66e9308638'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:44.819450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1387/2762] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:31:44 2022] DELETE /identity/v3/users/3bb2b5e770924b7383739a66e9308638 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:31:44.823884 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047cad19-2c52-44e1-8501-46f1fb02a63e None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:44.824408 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047cad19-2c52-44e1-8501-46f1fb02a63e None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:44.824512 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047cad19-2c52-44e1-8501-46f1fb02a63e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:44.843459 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-047cad19-2c52-44e1-8501-46f1fb02a63e None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:31:44.844646 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-047cad19-2c52-44e1-8501-46f1fb02a63e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:44.902055 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1376/2763] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:31:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:31:44.912003 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db935ab0-f12b-4031-b23c-ce6223ee0cfb admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:44.913324 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-db935ab0-f12b-4031-b23c-ce6223ee0cfb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:44.968034 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db935ab0-f12b-4031-b23c-ce6223ee0cfb admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:44.971620 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c272c7f-9416-4a20-b710-3c752e7e3c5a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:44.972201 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c272c7f-9416-4a20-b710-3c752e7e3c5a service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:44.972473 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c272c7f-9416-4a20-b710-3c752e7e3c5a service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:44.972741 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c272c7f-9416-4a20-b710-3c752e7e3c5a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:44.981337 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c272c7f-9416-4a20-b710-3c752e7e3c5a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:44.982791 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c272c7f-9416-4a20-b710-3c752e7e3c5a service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:44.992492 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1388/2764] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:31:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:31:45.258205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047cad19-2c52-44e1-8501-46f1fb02a63e None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:45.259644 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-047cad19-2c52-44e1-8501-46f1fb02a63e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:45.272082 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047cad19-2c52-44e1-8501-46f1fb02a63e None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:45.276698 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2f5034-daec-4fa9-87a8-04f057f1f46b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:45.277693 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2f5034-daec-4fa9-87a8-04f057f1f46b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:45.277906 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2f5034-daec-4fa9-87a8-04f057f1f46b admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:45.278181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2f5034-daec-4fa9-87a8-04f057f1f46b admin admin] PATH_INFO: `/v3/projects/b86c8a654e564d8fbe35c070363144f1` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:45.279695 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2f5034-daec-4fa9-87a8-04f057f1f46b admin admin] RBAC: Authorizing `identity:delete_project(project_id=b86c8a654e564d8fbe35c070363144f1)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:45.281583 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2f5034-daec-4fa9-87a8-04f057f1f46b admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:45.315330 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=96,keystone:INSERT=2,keystone:DELETE=2 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:45.332546 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-dc2f5034-daec-4fa9-87a8-04f057f1f46b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b86c8a654e564d8fbe35c070363144f1', 'request_id': 'req-dc2f5034-daec-4fa9-87a8-04f057f1f46b'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:45.338161 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-dc2f5034-daec-4fa9-87a8-04f057f1f46b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b86c8a654e564d8fbe35c070363144f1', 'request_id': 'req-dc2f5034-daec-4fa9-87a8-04f057f1f46b'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:45.344316 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-dc2f5034-daec-4fa9-87a8-04f057f1f46b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b86c8a654e564d8fbe35c070363144f1', 'request_id': 'req-dc2f5034-daec-4fa9-87a8-04f057f1f46b'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:45.356901 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-dc2f5034-daec-4fa9-87a8-04f057f1f46b admin admin] The token cache is being invalidate because project b86c8a654e564d8fbe35c070363144f1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:31:45.357199 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-dc2f5034-daec-4fa9-87a8-04f057f1f46b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:45.357846 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-dc2f5034-daec-4fa9-87a8-04f057f1f46b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:45.359303 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1377/2765] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:31:45 2022] DELETE /identity/v3/projects/b86c8a654e564d8fbe35c070363144f1 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:31:45.364559 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1c04ac-0bb6-4915-9ea9-f71dba50db56 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:45.364925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1c04ac-0bb6-4915-9ea9-f71dba50db56 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:45.365263 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1c04ac-0bb6-4915-9ea9-f71dba50db56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:45.384859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-bf1c04ac-0bb6-4915-9ea9-f71dba50db56 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:31:45.386071 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-bf1c04ac-0bb6-4915-9ea9-f71dba50db56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:45.444088 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1389/2766] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:31:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:31:45.451655 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3a28af-8f91-4bcc-9400-883d2ee75e68 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:45.451950 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3a28af-8f91-4bcc-9400-883d2ee75e68 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:45.452230 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3a28af-8f91-4bcc-9400-883d2ee75e68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:45.473099 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-2b3a28af-8f91-4bcc-9400-883d2ee75e68 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:31:45.474324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-2b3a28af-8f91-4bcc-9400-883d2ee75e68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:45.483238 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1378/2767] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:31:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:31:45.487341 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1c04ac-0bb6-4915-9ea9-f71dba50db56 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:45.496857 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1c04ac-0bb6-4915-9ea9-f71dba50db56 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:45.501348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7c6e52-f4e6-40bf-90c7-0b6e63556aab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:45.501971 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7c6e52-f4e6-40bf-90c7-0b6e63556aab admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:45.502250 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7c6e52-f4e6-40bf-90c7-0b6e63556aab admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:45.502522 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7c6e52-f4e6-40bf-90c7-0b6e63556aab admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:45.503730 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7c6e52-f4e6-40bf-90c7-0b6e63556aab admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:31:45.504031 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7c6e52-f4e6-40bf-90c7-0b6e63556aab admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:45.505644 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7c6e52-f4e6-40bf-90c7-0b6e63556aab admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:45.517069 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1390/2768] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:31:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:31:45.521728 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3a28af-8f91-4bcc-9400-883d2ee75e68 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:45.531072 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3a28af-8f91-4bcc-9400-883d2ee75e68 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:45.535639 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6c2112-e5e0-49b6-8a53-09700c937b5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:45.536225 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6c2112-e5e0-49b6-8a53-09700c937b5f admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:45.536500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6c2112-e5e0-49b6-8a53-09700c937b5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:45.536770 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6c2112-e5e0-49b6-8a53-09700c937b5f admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:45.537398 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6c2112-e5e0-49b6-8a53-09700c937b5f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:45.539287 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6c2112-e5e0-49b6-8a53-09700c937b5f admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:45.564985 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1379/2769] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:31:45 2022] POST /identity/v3/projects => generated 377 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:31:45.569471 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7c6e52-f4e6-40bf-90c7-0b6e63556aab admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:45.579317 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7c6e52-f4e6-40bf-90c7-0b6e63556aab admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:45.587519 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:45.637300 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2b980e-0066-4284-bace-dfbc848e0361 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:45.637922 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2b980e-0066-4284-bace-dfbc848e0361 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:45.638198 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2b980e-0066-4284-bace-dfbc848e0361 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:45.638463 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2b980e-0066-4284-bace-dfbc848e0361 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:45.639212 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2b980e-0066-4284-bace-dfbc848e0361 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:45.641280 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2b980e-0066-4284-bace-dfbc848e0361 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:45.665980 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1391/2770] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:31:45 2022] POST /identity/v3/users => generated 289 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:31:45.667150 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6c2112-e5e0-49b6-8a53-09700c937b5f admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:45.677236 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6c2112-e5e0-49b6-8a53-09700c937b5f admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:45.681699 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6460f3-85fb-4938-a139-77c9a8372dd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:45.682310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6460f3-85fb-4938-a139-77c9a8372dd5 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:45.682583 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6460f3-85fb-4938-a139-77c9a8372dd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:45.682851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6460f3-85fb-4938-a139-77c9a8372dd5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:45.683480 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb6460f3-85fb-4938-a139-77c9a8372dd5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:45.685027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb6460f3-85fb-4938-a139-77c9a8372dd5 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:45.695555 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1380/2771] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:31:45 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:31:45.699404 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2b980e-0066-4284-bace-dfbc848e0361 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:45.709599 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2b980e-0066-4284-bace-dfbc848e0361 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:45.714227 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4026d2c1-f314-470b-9825-4bb6b27dd6fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:45.714825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4026d2c1-f314-470b-9825-4bb6b27dd6fb admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:45.715102 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4026d2c1-f314-470b-9825-4bb6b27dd6fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:45.715395 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4026d2c1-f314-470b-9825-4bb6b27dd6fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:45.716027 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4026d2c1-f314-470b-9825-4bb6b27dd6fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:45.717596 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4026d2c1-f314-470b-9825-4bb6b27dd6fb admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:45.728063 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1392/2772] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:31:45 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:31:45.732125 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6460f3-85fb-4938-a139-77c9a8372dd5 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:45.742671 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6460f3-85fb-4938-a139-77c9a8372dd5 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:45.747234 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ad13e8-7c15-4189-9d40-ed13a3f505fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:45.747867 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ad13e8-7c15-4189-9d40-ed13a3f505fe admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:45.748144 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ad13e8-7c15-4189-9d40-ed13a3f505fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:45.748412 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ad13e8-7c15-4189-9d40-ed13a3f505fe admin admin] PATH_INFO: `/v3/projects/d64dbf9717824ea0b2299081aa8cb230/users/eb0afc1585e6438db3b1d62405ebdb47/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:45.763149 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ad13e8-7c15-4189-9d40-ed13a3f505fe admin admin] RBAC: Authorizing `identity:create_grant(project_id=d64dbf9717824ea0b2299081aa8cb230, user_id=eb0afc1585e6438db3b1d62405ebdb47, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:45.765714 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ad13e8-7c15-4189-9d40-ed13a3f505fe admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:45.778211 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1381/2773] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:31:45 2022] PUT /identity/v3/projects/d64dbf9717824ea0b2299081aa8cb230/users/eb0afc1585e6438db3b1d62405ebdb47/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:31:45.788376 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0a38aa-6140-4ff4-8e23-6614cfd78485 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:45.788690 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0a38aa-6140-4ff4-8e23-6614cfd78485 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:45.788965 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0a38aa-6140-4ff4-8e23-6614cfd78485 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:45.816647 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-cd0a38aa-6140-4ff4-8e23-6614cfd78485 None None] MFA Rules not processed for user `eb0afc1585e6438db3b1d62405ebdb47`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:31:45.817862 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-cd0a38aa-6140-4ff4-8e23-6614cfd78485 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:45.879267 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1393/2774] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:31:45 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:31:45.883623 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ad13e8-7c15-4189-9d40-ed13a3f505fe admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:45.942415 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ad13e8-7c15-4189-9d40-ed13a3f505fe admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:45.946946 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3220b262-7ff5-48e4-8b5e-229f73756984 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:45.947470 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3220b262-7ff5-48e4-8b5e-229f73756984 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:45.947744 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3220b262-7ff5-48e4-8b5e-229f73756984 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:45.948009 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3220b262-7ff5-48e4-8b5e-229f73756984 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:45.948618 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3220b262-7ff5-48e4-8b5e-229f73756984 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:45.950488 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3220b262-7ff5-48e4-8b5e-229f73756984 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:45.975421 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1382/2775] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:31:45 2022] POST /identity/v3/projects => generated 379 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:31:45.979563 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0a38aa-6140-4ff4-8e23-6614cfd78485 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:45.990394 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0a38aa-6140-4ff4-8e23-6614cfd78485 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:46.044477 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac3c854-3263-4f2b-be9d-636cfdfde578 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:46.045070 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac3c854-3263-4f2b-be9d-636cfdfde578 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:46.045346 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac3c854-3263-4f2b-be9d-636cfdfde578 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:46.045616 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac3c854-3263-4f2b-be9d-636cfdfde578 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:46.046267 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac3c854-3263-4f2b-be9d-636cfdfde578 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:46.048160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac3c854-3263-4f2b-be9d-636cfdfde578 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:46.069578 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1394/2776] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:31:45 2022] POST /identity/v3/users => generated 290 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:31:46.073506 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3220b262-7ff5-48e4-8b5e-229f73756984 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:46.083550 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3220b262-7ff5-48e4-8b5e-229f73756984 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:46.088021 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a0e526-c38d-4b49-809a-bc3ee147cf04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:46.088608 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a0e526-c38d-4b49-809a-bc3ee147cf04 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:46.088886 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a0e526-c38d-4b49-809a-bc3ee147cf04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:46.089154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a0e526-c38d-4b49-809a-bc3ee147cf04 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:46.089812 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a0e526-c38d-4b49-809a-bc3ee147cf04 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:46.091379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a0e526-c38d-4b49-809a-bc3ee147cf04 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:46.102111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1383/2777] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:31:46 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:31:46.105972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac3c854-3263-4f2b-be9d-636cfdfde578 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:46.116079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac3c854-3263-4f2b-be9d-636cfdfde578 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:46.120649 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbeec157-9c78-410f-8e94-b745584793dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:46.121235 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbeec157-9c78-410f-8e94-b745584793dc admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:46.121502 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbeec157-9c78-410f-8e94-b745584793dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:46.121794 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbeec157-9c78-410f-8e94-b745584793dc admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:46.122425 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbeec157-9c78-410f-8e94-b745584793dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:46.123982 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbeec157-9c78-410f-8e94-b745584793dc admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:46.126043 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:46.135343 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1395/2778] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:31:46 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:31:46.139264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a0e526-c38d-4b49-809a-bc3ee147cf04 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:46.149185 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a0e526-c38d-4b49-809a-bc3ee147cf04 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:46.154169 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d62277-0e4c-4243-a2fa-e4da43a32785 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:46.154802 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d62277-0e4c-4243-a2fa-e4da43a32785 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:46.155079 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d62277-0e4c-4243-a2fa-e4da43a32785 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:46.155349 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d62277-0e4c-4243-a2fa-e4da43a32785 admin admin] PATH_INFO: `/v3/projects/49c68419968342f5aba58695e36ba599/users/8d206e1c315c4e9db82511dd62a200f4/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:46.170325 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d62277-0e4c-4243-a2fa-e4da43a32785 admin admin] RBAC: Authorizing `identity:create_grant(project_id=49c68419968342f5aba58695e36ba599, user_id=8d206e1c315c4e9db82511dd62a200f4, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:46.173133 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d62277-0e4c-4243-a2fa-e4da43a32785 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:46.186192 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1384/2779] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:31:46 2022] PUT /identity/v3/projects/49c68419968342f5aba58695e36ba599/users/8d206e1c315c4e9db82511dd62a200f4/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:31:46.197605 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dd1a2c-68f3-43dd-a959-6f4e02fbd87b None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:46.197860 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dd1a2c-68f3-43dd-a959-6f4e02fbd87b None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:46.198131 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dd1a2c-68f3-43dd-a959-6f4e02fbd87b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:46.225682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-28dd1a2c-68f3-43dd-a959-6f4e02fbd87b None None] MFA Rules not processed for user `8d206e1c315c4e9db82511dd62a200f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:31:46.226899 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-28dd1a2c-68f3-43dd-a959-6f4e02fbd87b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:46.289412 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1396/2780] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:31:46 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:31:46.294006 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d62277-0e4c-4243-a2fa-e4da43a32785 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:46.298585 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=90,keystone:DELETE=5,keystone:INSERT=5 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:46.356345 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d62277-0e4c-4243-a2fa-e4da43a32785 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:46.363447 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf35e3a1-1a97-437d-93de-50e6c4db4acc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:46.364042 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf35e3a1-1a97-437d-93de-50e6c4db4acc admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:46.364317 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf35e3a1-1a97-437d-93de-50e6c4db4acc admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:46.364585 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf35e3a1-1a97-437d-93de-50e6c4db4acc admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:46.365217 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf35e3a1-1a97-437d-93de-50e6c4db4acc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:46.367107 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf35e3a1-1a97-437d-93de-50e6c4db4acc admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:46.392280 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1385/2781] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:31:46 2022] POST /identity/v3/projects => generated 377 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:31:46.396326 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28dd1a2c-68f3-43dd-a959-6f4e02fbd87b None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:46.407160 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28dd1a2c-68f3-43dd-a959-6f4e02fbd87b None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:46.462873 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2feb8592-5d6c-4cea-b6e6-f19d567def38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:46.463383 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2feb8592-5d6c-4cea-b6e6-f19d567def38 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:46.463661 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2feb8592-5d6c-4cea-b6e6-f19d567def38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:46.463929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2feb8592-5d6c-4cea-b6e6-f19d567def38 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:46.464553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2feb8592-5d6c-4cea-b6e6-f19d567def38 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:46.466478 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2feb8592-5d6c-4cea-b6e6-f19d567def38 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:46.486591 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1397/2782] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:31:46 2022] POST /identity/v3/users => generated 295 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:31:46.491186 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf35e3a1-1a97-437d-93de-50e6c4db4acc admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:46.501116 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf35e3a1-1a97-437d-93de-50e6c4db4acc admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:46.505651 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fe959d-99d9-4647-8096-f5a243bf35af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:46.506315 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fe959d-99d9-4647-8096-f5a243bf35af admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:46.506590 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fe959d-99d9-4647-8096-f5a243bf35af admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:46.506858 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fe959d-99d9-4647-8096-f5a243bf35af admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:46.507490 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fe959d-99d9-4647-8096-f5a243bf35af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:46.509038 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fe959d-99d9-4647-8096-f5a243bf35af admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:46.519764 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1386/2783] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:31:46 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:31:46.523594 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2feb8592-5d6c-4cea-b6e6-f19d567def38 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:46.533432 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2feb8592-5d6c-4cea-b6e6-f19d567def38 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:46.538897 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318328b4-5a11-4c5d-8f11-d30d4d556843 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:46.539745 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318328b4-5a11-4c5d-8f11-d30d4d556843 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:46.540027 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318328b4-5a11-4c5d-8f11-d30d4d556843 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:46.540299 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318328b4-5a11-4c5d-8f11-d30d4d556843 admin admin] PATH_INFO: `/v3/projects/9b4b56408d05473c8d33404de411de7b/users/f10dedb39c7e458790b0ace9b68160ff/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:46.554555 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318328b4-5a11-4c5d-8f11-d30d4d556843 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9b4b56408d05473c8d33404de411de7b, user_id=f10dedb39c7e458790b0ace9b68160ff, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:46.557304 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318328b4-5a11-4c5d-8f11-d30d4d556843 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:46.569406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1398/2784] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:31:46 2022] PUT /identity/v3/projects/9b4b56408d05473c8d33404de411de7b/users/f10dedb39c7e458790b0ace9b68160ff/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:31:46.573288 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fe959d-99d9-4647-8096-f5a243bf35af admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:46.583470 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fe959d-99d9-4647-8096-f5a243bf35af admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:46.637092 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0973398-6dc2-4e3a-87ea-01b716ec20c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:46.637675 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0973398-6dc2-4e3a-87ea-01b716ec20c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:46.637976 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0973398-6dc2-4e3a-87ea-01b716ec20c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:46.638245 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0973398-6dc2-4e3a-87ea-01b716ec20c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:46.638869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0973398-6dc2-4e3a-87ea-01b716ec20c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:46.640419 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0973398-6dc2-4e3a-87ea-01b716ec20c1 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:46.650179 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1387/2785] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:31:46 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:31:46.654449 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318328b4-5a11-4c5d-8f11-d30d4d556843 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:46.664896 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318328b4-5a11-4c5d-8f11-d30d4d556843 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:46.669399 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9604875-49c3-40f7-966b-2c42eafc4ae5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:46.670078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9604875-49c3-40f7-966b-2c42eafc4ae5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:46.670356 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9604875-49c3-40f7-966b-2c42eafc4ae5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:46.670785 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9604875-49c3-40f7-966b-2c42eafc4ae5 admin admin] PATH_INFO: `/v3/projects/9b4b56408d05473c8d33404de411de7b/users/f10dedb39c7e458790b0ace9b68160ff/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:46.673019 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9604875-49c3-40f7-966b-2c42eafc4ae5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9b4b56408d05473c8d33404de411de7b, user_id=f10dedb39c7e458790b0ace9b68160ff, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:46.675659 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9604875-49c3-40f7-966b-2c42eafc4ae5 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:46.685647 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1399/2786] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:31:46 2022] PUT /identity/v3/projects/9b4b56408d05473c8d33404de411de7b/users/f10dedb39c7e458790b0ace9b68160ff/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:31:46.696067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a77bca-dc7d-444e-ad6e-d319df4366c8 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:46.696351 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a77bca-dc7d-444e-ad6e-d319df4366c8 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:46.696624 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a77bca-dc7d-444e-ad6e-d319df4366c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:46.724537 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-e7a77bca-dc7d-444e-ad6e-d319df4366c8 None None] MFA Rules not processed for user `f10dedb39c7e458790b0ace9b68160ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:31:46.726006 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-e7a77bca-dc7d-444e-ad6e-d319df4366c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:46.787340 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:46.793534 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1388/2787] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:31:46 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:31:46.804575 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9604875-49c3-40f7-966b-2c42eafc4ae5 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:46.806321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d9604875-49c3-40f7-966b-2c42eafc4ae5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:46.848573 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:46.883395 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9604875-49c3-40f7-966b-2c42eafc4ae5 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:46.886967 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e056a9d-5805-4a42-a6df-7362fdb81591 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:46.887555 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e056a9d-5805-4a42-a6df-7362fdb81591 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:46.887825 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e056a9d-5805-4a42-a6df-7362fdb81591 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:46.888089 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e056a9d-5805-4a42-a6df-7362fdb81591 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:46.897130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e056a9d-5805-4a42-a6df-7362fdb81591 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:46.898557 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e056a9d-5805-4a42-a6df-7362fdb81591 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:46.907817 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1400/2788] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:31:46 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:31:46.948660 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a77bca-dc7d-444e-ad6e-d319df4366c8 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:46.950085 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-e7a77bca-dc7d-444e-ad6e-d319df4366c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:46.962352 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a77bca-dc7d-444e-ad6e-d319df4366c8 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:46.966200 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe46c6c7-2025-4f09-a4af-02f16f8e5081 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:46.966790 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe46c6c7-2025-4f09-a4af-02f16f8e5081 service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:46.967070 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe46c6c7-2025-4f09-a4af-02f16f8e5081 service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:46.967342 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe46c6c7-2025-4f09-a4af-02f16f8e5081 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:47.022998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe46c6c7-2025-4f09-a4af-02f16f8e5081 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:47.024410 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe46c6c7-2025-4f09-a4af-02f16f8e5081 service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:47.033272 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1389/2789] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:31:46 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:31:49.904074 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e056a9d-5805-4a42-a6df-7362fdb81591 service cinder] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:49.965481 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e056a9d-5805-4a42-a6df-7362fdb81591 service cinder] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:49.969986 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8c4919-d1c8-4315-bafd-e0cb14c22e51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:49.971106 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8c4919-d1c8-4315-bafd-e0cb14c22e51 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:49.971411 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8c4919-d1c8-4315-bafd-e0cb14c22e51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:49.971689 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8c4919-d1c8-4315-bafd-e0cb14c22e51 admin admin] PATH_INFO: `/v3/users/eb0afc1585e6438db3b1d62405ebdb47` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:49.972787 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8c4919-d1c8-4315-bafd-e0cb14c22e51 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eb0afc1585e6438db3b1d62405ebdb47)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:49.974725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8c4919-d1c8-4315-bafd-e0cb14c22e51 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:50.020890 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-be8c4919-d1c8-4315-bafd-e0cb14c22e51 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eb0afc1585e6438db3b1d62405ebdb47'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:50.027914 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-be8c4919-d1c8-4315-bafd-e0cb14c22e51 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eb0afc1585e6438db3b1d62405ebdb47'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:50.028565 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-be8c4919-d1c8-4315-bafd-e0cb14c22e51 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eb0afc1585e6438db3b1d62405ebdb47'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:50.035785 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-be8c4919-d1c8-4315-bafd-e0cb14c22e51 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eb0afc1585e6438db3b1d62405ebdb47'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:50.047919 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-be8c4919-d1c8-4315-bafd-e0cb14c22e51 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eb0afc1585e6438db3b1d62405ebdb47'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:50.053846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1401/2790] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:31:49 2022] DELETE /identity/v3/users/eb0afc1585e6438db3b1d62405ebdb47 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:31:50.058244 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3dd138-2be6-4ca0-b368-37ecc85fe8eb None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:50.058770 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3dd138-2be6-4ca0-b368-37ecc85fe8eb None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:50.058883 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3dd138-2be6-4ca0-b368-37ecc85fe8eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:50.079015 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-af3dd138-2be6-4ca0-b368-37ecc85fe8eb None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:31:50.080213 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-af3dd138-2be6-4ca0-b368-37ecc85fe8eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:50.136577 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1390/2791] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:31:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:31:50.146567 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8c4919-d1c8-4315-bafd-e0cb14c22e51 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:50.147937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-be8c4919-d1c8-4315-bafd-e0cb14c22e51 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:50.192977 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:50.206235 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8c4919-d1c8-4315-bafd-e0cb14c22e51 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:50.210023 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9404a66-4fd5-44b8-9611-86c2a978ca98 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:50.210617 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9404a66-4fd5-44b8-9611-86c2a978ca98 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:50.210893 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9404a66-4fd5-44b8-9611-86c2a978ca98 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:50.211163 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9404a66-4fd5-44b8-9611-86c2a978ca98 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:50.220049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9404a66-4fd5-44b8-9611-86c2a978ca98 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:50.222024 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9404a66-4fd5-44b8-9611-86c2a978ca98 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:50.231211 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1402/2792] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:31:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:31:50.492964 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af3dd138-2be6-4ca0-b368-37ecc85fe8eb None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:50.494407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-af3dd138-2be6-4ca0-b368-37ecc85fe8eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:50.506229 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af3dd138-2be6-4ca0-b368-37ecc85fe8eb None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:50.511158 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e103bec1-d3b6-43d2-976c-31a7c311fad3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:50.511791 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e103bec1-d3b6-43d2-976c-31a7c311fad3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:50.512067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e103bec1-d3b6-43d2-976c-31a7c311fad3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:50.512339 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e103bec1-d3b6-43d2-976c-31a7c311fad3 admin admin] PATH_INFO: `/v3/projects/d64dbf9717824ea0b2299081aa8cb230` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:50.514030 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e103bec1-d3b6-43d2-976c-31a7c311fad3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d64dbf9717824ea0b2299081aa8cb230)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:50.517120 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e103bec1-d3b6-43d2-976c-31a7c311fad3 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:50.563973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-e103bec1-d3b6-43d2-976c-31a7c311fad3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd64dbf9717824ea0b2299081aa8cb230', 'request_id': 'req-e103bec1-d3b6-43d2-976c-31a7c311fad3'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:50.569393 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-e103bec1-d3b6-43d2-976c-31a7c311fad3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd64dbf9717824ea0b2299081aa8cb230', 'request_id': 'req-e103bec1-d3b6-43d2-976c-31a7c311fad3'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:50.575324 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-e103bec1-d3b6-43d2-976c-31a7c311fad3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd64dbf9717824ea0b2299081aa8cb230', 'request_id': 'req-e103bec1-d3b6-43d2-976c-31a7c311fad3'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:50.587240 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-e103bec1-d3b6-43d2-976c-31a7c311fad3 admin admin] The token cache is being invalidate because project d64dbf9717824ea0b2299081aa8cb230 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:31:50.587529 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-e103bec1-d3b6-43d2-976c-31a7c311fad3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:50.588174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-e103bec1-d3b6-43d2-976c-31a7c311fad3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:50.589656 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1391/2793] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:31:50 2022] DELETE /identity/v3/projects/d64dbf9717824ea0b2299081aa8cb230 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:31:50.593601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9404a66-4fd5-44b8-9611-86c2a978ca98 service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:50.603541 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9404a66-4fd5-44b8-9611-86c2a978ca98 service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:50.657417 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b1f48a-83b5-4515-9af7-fd3f39205016 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:50.658011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b1f48a-83b5-4515-9af7-fd3f39205016 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:50.658284 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b1f48a-83b5-4515-9af7-fd3f39205016 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:50.658548 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b1f48a-83b5-4515-9af7-fd3f39205016 admin admin] PATH_INFO: `/v3/users/8d206e1c315c4e9db82511dd62a200f4` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:50.659625 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b1f48a-83b5-4515-9af7-fd3f39205016 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8d206e1c315c4e9db82511dd62a200f4)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:50.661501 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b1f48a-83b5-4515-9af7-fd3f39205016 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:50.706642 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-e9b1f48a-83b5-4515-9af7-fd3f39205016 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8d206e1c315c4e9db82511dd62a200f4'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:50.713313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-e9b1f48a-83b5-4515-9af7-fd3f39205016 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8d206e1c315c4e9db82511dd62a200f4'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:50.714000 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-e9b1f48a-83b5-4515-9af7-fd3f39205016 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8d206e1c315c4e9db82511dd62a200f4'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:50.721045 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-e9b1f48a-83b5-4515-9af7-fd3f39205016 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8d206e1c315c4e9db82511dd62a200f4'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:50.732882 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-e9b1f48a-83b5-4515-9af7-fd3f39205016 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8d206e1c315c4e9db82511dd62a200f4'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:50.738835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1403/2794] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:31:50 2022] DELETE /identity/v3/users/8d206e1c315c4e9db82511dd62a200f4 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:31:51.018543 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e103bec1-d3b6-43d2-976c-31a7c311fad3 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:51.024002 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-e103bec1-d3b6-43d2-976c-31a7c311fad3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:51.039638 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e103bec1-d3b6-43d2-976c-31a7c311fad3 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:51.056683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:51.097430 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4711641e-e05f-49ec-aba7-04c8049a32d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:51.098161 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4711641e-e05f-49ec-aba7-04c8049a32d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:51.098358 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4711641e-e05f-49ec-aba7-04c8049a32d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:51.098629 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4711641e-e05f-49ec-aba7-04c8049a32d0 admin admin] PATH_INFO: `/v3/projects/49c68419968342f5aba58695e36ba599` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:51.099755 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4711641e-e05f-49ec-aba7-04c8049a32d0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=49c68419968342f5aba58695e36ba599)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:51.101638 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4711641e-e05f-49ec-aba7-04c8049a32d0 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:51.149017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-4711641e-e05f-49ec-aba7-04c8049a32d0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '49c68419968342f5aba58695e36ba599', 'request_id': 'req-4711641e-e05f-49ec-aba7-04c8049a32d0'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:51.154939 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-4711641e-e05f-49ec-aba7-04c8049a32d0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '49c68419968342f5aba58695e36ba599', 'request_id': 'req-4711641e-e05f-49ec-aba7-04c8049a32d0'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:51.161286 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-4711641e-e05f-49ec-aba7-04c8049a32d0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '49c68419968342f5aba58695e36ba599', 'request_id': 'req-4711641e-e05f-49ec-aba7-04c8049a32d0'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:51.174219 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-4711641e-e05f-49ec-aba7-04c8049a32d0 admin admin] The token cache is being invalidate because project 49c68419968342f5aba58695e36ba599 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:31:51.174506 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-4711641e-e05f-49ec-aba7-04c8049a32d0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:51.175142 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-4711641e-e05f-49ec-aba7-04c8049a32d0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:51.176621 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1392/2795] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:31:51 2022] DELETE /identity/v3/projects/49c68419968342f5aba58695e36ba599 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:31:51.180659 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b1f48a-83b5-4515-9af7-fd3f39205016 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:51.182148 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-e9b1f48a-83b5-4515-9af7-fd3f39205016 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:51.195319 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b1f48a-83b5-4515-9af7-fd3f39205016 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:51.249775 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4378127-d078-42aa-a1e9-7102cf0a076a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:51.250401 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4378127-d078-42aa-a1e9-7102cf0a076a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:51.250674 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4378127-d078-42aa-a1e9-7102cf0a076a admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:51.250942 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4378127-d078-42aa-a1e9-7102cf0a076a admin admin] PATH_INFO: `/v3/users/f10dedb39c7e458790b0ace9b68160ff` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:51.252556 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4378127-d078-42aa-a1e9-7102cf0a076a admin admin] RBAC: Authorizing `identity:delete_user(user_id=f10dedb39c7e458790b0ace9b68160ff)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:51.254475 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4378127-d078-42aa-a1e9-7102cf0a076a admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:51.258800 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:51.300801 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a4378127-d078-42aa-a1e9-7102cf0a076a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f10dedb39c7e458790b0ace9b68160ff'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:51.307875 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a4378127-d078-42aa-a1e9-7102cf0a076a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f10dedb39c7e458790b0ace9b68160ff'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:51.308535 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a4378127-d078-42aa-a1e9-7102cf0a076a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f10dedb39c7e458790b0ace9b68160ff'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:51.316301 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a4378127-d078-42aa-a1e9-7102cf0a076a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f10dedb39c7e458790b0ace9b68160ff'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:51.328276 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a4378127-d078-42aa-a1e9-7102cf0a076a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f10dedb39c7e458790b0ace9b68160ff'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:51.334117 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1404/2796] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:31:51 2022] DELETE /identity/v3/users/f10dedb39c7e458790b0ace9b68160ff => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:31:51.603157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4711641e-e05f-49ec-aba7-04c8049a32d0 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:51.604630 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-4711641e-e05f-49ec-aba7-04c8049a32d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:51.621373 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4711641e-e05f-49ec-aba7-04c8049a32d0 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:51.678101 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a41423d-cbfe-49e1-a73a-5813df8ca11e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:51.678735 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a41423d-cbfe-49e1-a73a-5813df8ca11e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:51.679015 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a41423d-cbfe-49e1-a73a-5813df8ca11e admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:51.679308 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a41423d-cbfe-49e1-a73a-5813df8ca11e admin admin] PATH_INFO: `/v3/projects/9b4b56408d05473c8d33404de411de7b` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:51.680429 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a41423d-cbfe-49e1-a73a-5813df8ca11e admin admin] RBAC: Authorizing `identity:delete_project(project_id=9b4b56408d05473c8d33404de411de7b)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:51.682340 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a41423d-cbfe-49e1-a73a-5813df8ca11e admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:51.726721 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-3a41423d-cbfe-49e1-a73a-5813df8ca11e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9b4b56408d05473c8d33404de411de7b', 'request_id': 'req-3a41423d-cbfe-49e1-a73a-5813df8ca11e'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:51.728615 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:51.733772 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-3a41423d-cbfe-49e1-a73a-5813df8ca11e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9b4b56408d05473c8d33404de411de7b', 'request_id': 'req-3a41423d-cbfe-49e1-a73a-5813df8ca11e'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:51.741563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-3a41423d-cbfe-49e1-a73a-5813df8ca11e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9b4b56408d05473c8d33404de411de7b', 'request_id': 'req-3a41423d-cbfe-49e1-a73a-5813df8ca11e'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:51.754078 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-3a41423d-cbfe-49e1-a73a-5813df8ca11e admin admin] The token cache is being invalidate because project 9b4b56408d05473c8d33404de411de7b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:31:51.754508 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-3a41423d-cbfe-49e1-a73a-5813df8ca11e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:51.755142 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-3a41423d-cbfe-49e1-a73a-5813df8ca11e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:51.756636 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1393/2797] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:31:51 2022] DELETE /identity/v3/projects/9b4b56408d05473c8d33404de411de7b => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:31:51.762522 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef2b88a-36a9-4d65-b962-f6096d52a4ef None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:51.762805 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef2b88a-36a9-4d65-b962-f6096d52a4ef None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:51.763079 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef2b88a-36a9-4d65-b962-f6096d52a4ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:51.783550 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-6ef2b88a-36a9-4d65-b962-f6096d52a4ef None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:31:51.784766 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6ef2b88a-36a9-4d65-b962-f6096d52a4ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:51.839972 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1405/2798] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:31:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:31:51.848978 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64f31ab-c719-4ec6-8848-9b3907955212 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:51.849260 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64f31ab-c719-4ec6-8848-9b3907955212 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:51.849533 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64f31ab-c719-4ec6-8848-9b3907955212 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:51.869089 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-a64f31ab-c719-4ec6-8848-9b3907955212 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:31:51.870367 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-a64f31ab-c719-4ec6-8848-9b3907955212 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:51.880564 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1394/2799] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:31:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:31:51.884518 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef2b88a-36a9-4d65-b962-f6096d52a4ef None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:51.894628 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef2b88a-36a9-4d65-b962-f6096d52a4ef None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:51.899099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2603bb5-26bc-4bff-be8b-caadf27209df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:51.899685 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2603bb5-26bc-4bff-be8b-caadf27209df admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:51.899959 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2603bb5-26bc-4bff-be8b-caadf27209df admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:51.900233 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2603bb5-26bc-4bff-be8b-caadf27209df admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:51.900868 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2603bb5-26bc-4bff-be8b-caadf27209df admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:31:51.901415 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2603bb5-26bc-4bff-be8b-caadf27209df admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:51.903023 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2603bb5-26bc-4bff-be8b-caadf27209df admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:51.915030 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1406/2800] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:31:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 04:31:51.919459 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64f31ab-c719-4ec6-8848-9b3907955212 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:51.928482 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64f31ab-c719-4ec6-8848-9b3907955212 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:51.932973 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4614b41-d4b7-477e-8a5b-620c73034189 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:51.933546 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4614b41-d4b7-477e-8a5b-620c73034189 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:51.933859 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4614b41-d4b7-477e-8a5b-620c73034189 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:51.934132 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4614b41-d4b7-477e-8a5b-620c73034189 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:51.934754 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4614b41-d4b7-477e-8a5b-620c73034189 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:51.936583 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4614b41-d4b7-477e-8a5b-620c73034189 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:51.963130 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1395/2801] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:31:51 2022] POST /identity/v3/projects => generated 381 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:31:51.967028 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2603bb5-26bc-4bff-be8b-caadf27209df admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:51.977437 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2603bb5-26bc-4bff-be8b-caadf27209df admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:52.031474 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc0a688-b715-4544-9db6-5c81da17b09a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:52.031991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc0a688-b715-4544-9db6-5c81da17b09a admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:52.032265 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc0a688-b715-4544-9db6-5c81da17b09a admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:52.032534 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc0a688-b715-4544-9db6-5c81da17b09a admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:52.033709 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc0a688-b715-4544-9db6-5c81da17b09a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:52.035612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc0a688-b715-4544-9db6-5c81da17b09a admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:52.054411 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:52.061862 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1407/2802] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:31:51 2022] POST /identity/v3/users => generated 291 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:31:52.066185 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4614b41-d4b7-477e-8a5b-620c73034189 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:52.076176 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4614b41-d4b7-477e-8a5b-620c73034189 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:52.081186 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26421f1-359e-47ab-9886-3aaf57b698fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:52.081792 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26421f1-359e-47ab-9886-3aaf57b698fb admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:52.082072 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26421f1-359e-47ab-9886-3aaf57b698fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:52.082341 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26421f1-359e-47ab-9886-3aaf57b698fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:52.082969 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26421f1-359e-47ab-9886-3aaf57b698fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:52.084528 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26421f1-359e-47ab-9886-3aaf57b698fb admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:52.094103 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1396/2803] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:31:52 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:31:52.098556 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc0a688-b715-4544-9db6-5c81da17b09a admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:52.108332 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc0a688-b715-4544-9db6-5c81da17b09a admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:52.112859 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95587e7d-4c3e-407e-a8f5-61ab719763b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:52.113447 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95587e7d-4c3e-407e-a8f5-61ab719763b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:52.113720 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95587e7d-4c3e-407e-a8f5-61ab719763b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:52.114016 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95587e7d-4c3e-407e-a8f5-61ab719763b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:52.114672 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95587e7d-4c3e-407e-a8f5-61ab719763b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:52.116255 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95587e7d-4c3e-407e-a8f5-61ab719763b5 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:52.127130 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1408/2804] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:31:52 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:31:52.133728 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26421f1-359e-47ab-9886-3aaf57b698fb admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:52.144278 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26421f1-359e-47ab-9886-3aaf57b698fb admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:52.148864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27de336-60ad-4047-b8f5-4bc0d0af9c50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:52.149538 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27de336-60ad-4047-b8f5-4bc0d0af9c50 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:52.149847 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27de336-60ad-4047-b8f5-4bc0d0af9c50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:52.150119 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27de336-60ad-4047-b8f5-4bc0d0af9c50 admin admin] PATH_INFO: `/v3/projects/b90ca78e4258450eb59c8bcd7053c118/users/827da44f87d44627866be41756b9e821/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:52.164853 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27de336-60ad-4047-b8f5-4bc0d0af9c50 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b90ca78e4258450eb59c8bcd7053c118, user_id=827da44f87d44627866be41756b9e821, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:52.167470 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27de336-60ad-4047-b8f5-4bc0d0af9c50 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:52.179571 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1397/2805] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:31:52 2022] PUT /identity/v3/projects/b90ca78e4258450eb59c8bcd7053c118/users/827da44f87d44627866be41756b9e821/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:31:52.243298 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc08c173-fc14-49cd-a053-2f6b29cc96e8 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:52.243887 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc08c173-fc14-49cd-a053-2f6b29cc96e8 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:52.243887 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc08c173-fc14-49cd-a053-2f6b29cc96e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:52.273388 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-fc08c173-fc14-49cd-a053-2f6b29cc96e8 None None] MFA Rules not processed for user `827da44f87d44627866be41756b9e821`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:31:52.274628 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-fc08c173-fc14-49cd-a053-2f6b29cc96e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:52.336064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1409/2806] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:31:52 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:31:52.340441 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27de336-60ad-4047-b8f5-4bc0d0af9c50 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:52.400925 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27de336-60ad-4047-b8f5-4bc0d0af9c50 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:52.405786 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0538f02e-08c5-43c6-b255-031255d5ff4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:52.406390 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0538f02e-08c5-43c6-b255-031255d5ff4b admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:52.406665 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0538f02e-08c5-43c6-b255-031255d5ff4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:52.406936 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0538f02e-08c5-43c6-b255-031255d5ff4b admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:52.407558 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0538f02e-08c5-43c6-b255-031255d5ff4b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:52.409457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0538f02e-08c5-43c6-b255-031255d5ff4b admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:52.435274 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1398/2807] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:31:52 2022] POST /identity/v3/projects => generated 381 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:31:52.439910 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc08c173-fc14-49cd-a053-2f6b29cc96e8 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:52.450636 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc08c173-fc14-49cd-a053-2f6b29cc96e8 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:52.507003 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60ca163-57cd-4508-be97-7f4883936feb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:52.507600 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60ca163-57cd-4508-be97-7f4883936feb admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:52.507872 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60ca163-57cd-4508-be97-7f4883936feb admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:52.508137 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60ca163-57cd-4508-be97-7f4883936feb admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:52.508758 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60ca163-57cd-4508-be97-7f4883936feb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:52.510674 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60ca163-57cd-4508-be97-7f4883936feb admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:52.531117 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1410/2808] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:31:52 2022] POST /identity/v3/users => generated 297 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:31:52.535209 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0538f02e-08c5-43c6-b255-031255d5ff4b admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:52.545099 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0538f02e-08c5-43c6-b255-031255d5ff4b admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:52.550165 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251d8504-46f3-43ae-8a04-d4e01021281c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:52.550759 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251d8504-46f3-43ae-8a04-d4e01021281c admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:52.551035 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251d8504-46f3-43ae-8a04-d4e01021281c admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:52.551303 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251d8504-46f3-43ae-8a04-d4e01021281c admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:52.551937 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251d8504-46f3-43ae-8a04-d4e01021281c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:52.553494 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251d8504-46f3-43ae-8a04-d4e01021281c admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:52.563863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1399/2809] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:31:52 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:31:52.568484 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60ca163-57cd-4508-be97-7f4883936feb admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:52.578520 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60ca163-57cd-4508-be97-7f4883936feb admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:52.583104 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fc1928-bc21-4921-8119-3e0deed23cf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:52.583743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fc1928-bc21-4921-8119-3e0deed23cf3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:52.584021 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fc1928-bc21-4921-8119-3e0deed23cf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:52.584292 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fc1928-bc21-4921-8119-3e0deed23cf3 admin admin] PATH_INFO: `/v3/projects/59ce799c6ecc4e72819d87324d89a8c6/users/b2477b63b6e44ae79f3ee69a08601912/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:52.598478 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fc1928-bc21-4921-8119-3e0deed23cf3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=59ce799c6ecc4e72819d87324d89a8c6, user_id=b2477b63b6e44ae79f3ee69a08601912, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:52.601187 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fc1928-bc21-4921-8119-3e0deed23cf3 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:52.614522 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1411/2810] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:31:52 2022] PUT /identity/v3/projects/59ce799c6ecc4e72819d87324d89a8c6/users/b2477b63b6e44ae79f3ee69a08601912/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:31:52.618524 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251d8504-46f3-43ae-8a04-d4e01021281c admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:52.628864 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251d8504-46f3-43ae-8a04-d4e01021281c admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:52.675591 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:52.686801 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5268ed-91ee-4eb1-a252-d12a1e2a8c98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:52.687395 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5268ed-91ee-4eb1-a252-d12a1e2a8c98 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:52.687706 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5268ed-91ee-4eb1-a252-d12a1e2a8c98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:52.687978 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5268ed-91ee-4eb1-a252-d12a1e2a8c98 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:52.688612 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf5268ed-91ee-4eb1-a252-d12a1e2a8c98 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:52.690307 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf5268ed-91ee-4eb1-a252-d12a1e2a8c98 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:52.700164 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1400/2811] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:31:52 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:31:52.704285 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fc1928-bc21-4921-8119-3e0deed23cf3 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:52.715094 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fc1928-bc21-4921-8119-3e0deed23cf3 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:52.719593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889d65aa-6ae6-476f-bf92-80860a6364e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:52.720773 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889d65aa-6ae6-476f-bf92-80860a6364e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:52.721046 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889d65aa-6ae6-476f-bf92-80860a6364e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:52.721313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889d65aa-6ae6-476f-bf92-80860a6364e3 admin admin] PATH_INFO: `/v3/projects/59ce799c6ecc4e72819d87324d89a8c6/users/b2477b63b6e44ae79f3ee69a08601912/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:52.723230 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889d65aa-6ae6-476f-bf92-80860a6364e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=59ce799c6ecc4e72819d87324d89a8c6, user_id=b2477b63b6e44ae79f3ee69a08601912, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:52.725864 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889d65aa-6ae6-476f-bf92-80860a6364e3 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:52.735749 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1412/2812] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:31:52 2022] PUT /identity/v3/projects/59ce799c6ecc4e72819d87324d89a8c6/users/b2477b63b6e44ae79f3ee69a08601912/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:31:52.745999 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a219032c-8a86-4c79-bd23-14de8e8e3e07 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:52.746284 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a219032c-8a86-4c79-bd23-14de8e8e3e07 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:52.746566 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a219032c-8a86-4c79-bd23-14de8e8e3e07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:52.774104 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-a219032c-8a86-4c79-bd23-14de8e8e3e07 None None] MFA Rules not processed for user `b2477b63b6e44ae79f3ee69a08601912`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:31:52.775311 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-a219032c-8a86-4c79-bd23-14de8e8e3e07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:52.841151 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1401/2813] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:31:52 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:31:52.852548 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889d65aa-6ae6-476f-bf92-80860a6364e3 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:52.854237 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-889d65aa-6ae6-476f-bf92-80860a6364e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:52.887621 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:52.912175 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889d65aa-6ae6-476f-bf92-80860a6364e3 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:52.915824 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35299d1-d861-4719-87a4-ecf67fd7fece service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:52.916751 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35299d1-d861-4719-87a4-ecf67fd7fece service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:52.917233 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35299d1-d861-4719-87a4-ecf67fd7fece service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:52.917714 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35299d1-d861-4719-87a4-ecf67fd7fece service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:52.928355 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35299d1-d861-4719-87a4-ecf67fd7fece service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:52.929792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35299d1-d861-4719-87a4-ecf67fd7fece service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:52.939502 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1413/2814] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:31:52 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:31:53.333702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a219032c-8a86-4c79-bd23-14de8e8e3e07 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:53.403777 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a219032c-8a86-4c79-bd23-14de8e8e3e07 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:53.408211 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29de613c-54a5-44d0-9059-d08e4d6a8740 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:53.408840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29de613c-54a5-44d0-9059-d08e4d6a8740 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:53.409124 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29de613c-54a5-44d0-9059-d08e4d6a8740 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:53.409396 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29de613c-54a5-44d0-9059-d08e4d6a8740 admin admin] PATH_INFO: `/v3/users/827da44f87d44627866be41756b9e821` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:53.410569 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29de613c-54a5-44d0-9059-d08e4d6a8740 admin admin] RBAC: Authorizing `identity:delete_user(user_id=827da44f87d44627866be41756b9e821)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:53.412455 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29de613c-54a5-44d0-9059-d08e4d6a8740 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:53.457696 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-29de613c-54a5-44d0-9059-d08e4d6a8740 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '827da44f87d44627866be41756b9e821'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:53.464579 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-29de613c-54a5-44d0-9059-d08e4d6a8740 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '827da44f87d44627866be41756b9e821'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:53.465297 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-29de613c-54a5-44d0-9059-d08e4d6a8740 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '827da44f87d44627866be41756b9e821'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:53.472211 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-29de613c-54a5-44d0-9059-d08e4d6a8740 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '827da44f87d44627866be41756b9e821'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:53.479900 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:53.484895 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-29de613c-54a5-44d0-9059-d08e4d6a8740 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '827da44f87d44627866be41756b9e821'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:53.492387 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1402/2815] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:31:53 2022] DELETE /identity/v3/users/827da44f87d44627866be41756b9e821 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:31:53.496835 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc6998d-ba18-4eed-8704-fae20ccb8762 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:53.497299 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc6998d-ba18-4eed-8704-fae20ccb8762 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:53.497396 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc6998d-ba18-4eed-8704-fae20ccb8762 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:53.516409 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-6cc6998d-ba18-4eed-8704-fae20ccb8762 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:31:53.517633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6cc6998d-ba18-4eed-8704-fae20ccb8762 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:53.575364 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1414/2816] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:31:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:31:53.586869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29de613c-54a5-44d0-9059-d08e4d6a8740 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:53.588248 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-29de613c-54a5-44d0-9059-d08e4d6a8740 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:53.646076 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29de613c-54a5-44d0-9059-d08e4d6a8740 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:53.649673 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1ccfdd-3828-4c87-8aec-5dd415408c9c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:53.650302 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1ccfdd-3828-4c87-8aec-5dd415408c9c service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:53.650579 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1ccfdd-3828-4c87-8aec-5dd415408c9c service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:53.650847 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1ccfdd-3828-4c87-8aec-5dd415408c9c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:53.660709 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1ccfdd-3828-4c87-8aec-5dd415408c9c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:53.662088 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1ccfdd-3828-4c87-8aec-5dd415408c9c service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:53.671547 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1403/2817] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:31:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:31:53.947491 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc6998d-ba18-4eed-8704-fae20ccb8762 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:53.948975 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6cc6998d-ba18-4eed-8704-fae20ccb8762 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:53.962818 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc6998d-ba18-4eed-8704-fae20ccb8762 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:53.967556 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14938b9e-2ff1-4554-8121-c85015dca6d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:53.968190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14938b9e-2ff1-4554-8121-c85015dca6d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:53.968581 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14938b9e-2ff1-4554-8121-c85015dca6d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:53.968857 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14938b9e-2ff1-4554-8121-c85015dca6d0 admin admin] PATH_INFO: `/v3/projects/b90ca78e4258450eb59c8bcd7053c118` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:53.970667 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14938b9e-2ff1-4554-8121-c85015dca6d0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b90ca78e4258450eb59c8bcd7053c118)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:53.972544 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14938b9e-2ff1-4554-8121-c85015dca6d0 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:54.021820 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-14938b9e-2ff1-4554-8121-c85015dca6d0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b90ca78e4258450eb59c8bcd7053c118', 'request_id': 'req-14938b9e-2ff1-4554-8121-c85015dca6d0'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:54.027427 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-14938b9e-2ff1-4554-8121-c85015dca6d0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b90ca78e4258450eb59c8bcd7053c118', 'request_id': 'req-14938b9e-2ff1-4554-8121-c85015dca6d0'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:54.034040 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-14938b9e-2ff1-4554-8121-c85015dca6d0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b90ca78e4258450eb59c8bcd7053c118', 'request_id': 'req-14938b9e-2ff1-4554-8121-c85015dca6d0'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:54.046294 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-14938b9e-2ff1-4554-8121-c85015dca6d0 admin admin] The token cache is being invalidate because project b90ca78e4258450eb59c8bcd7053c118 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:31:54.046578 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-14938b9e-2ff1-4554-8121-c85015dca6d0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:54.047188 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-14938b9e-2ff1-4554-8121-c85015dca6d0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:54.048902 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1415/2818] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:31:53 2022] DELETE /identity/v3/projects/b90ca78e4258450eb59c8bcd7053c118 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:31:54.052684 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1ccfdd-3828-4c87-8aec-5dd415408c9c service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:54.062867 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1ccfdd-3828-4c87-8aec-5dd415408c9c service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:54.136264 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf43089b-5d5c-4156-8c37-d70415125d2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:54.136897 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf43089b-5d5c-4156-8c37-d70415125d2e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:54.137176 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf43089b-5d5c-4156-8c37-d70415125d2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:54.137474 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf43089b-5d5c-4156-8c37-d70415125d2e admin admin] PATH_INFO: `/v3/users/b2477b63b6e44ae79f3ee69a08601912` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:54.138635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf43089b-5d5c-4156-8c37-d70415125d2e admin admin] RBAC: Authorizing `identity:delete_user(user_id=b2477b63b6e44ae79f3ee69a08601912)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:54.140618 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf43089b-5d5c-4156-8c37-d70415125d2e admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:54.195933 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-cf43089b-5d5c-4156-8c37-d70415125d2e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b2477b63b6e44ae79f3ee69a08601912'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:54.206906 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-cf43089b-5d5c-4156-8c37-d70415125d2e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b2477b63b6e44ae79f3ee69a08601912'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:54.207610 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-cf43089b-5d5c-4156-8c37-d70415125d2e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b2477b63b6e44ae79f3ee69a08601912'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:54.215208 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-cf43089b-5d5c-4156-8c37-d70415125d2e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b2477b63b6e44ae79f3ee69a08601912'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:54.228874 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-cf43089b-5d5c-4156-8c37-d70415125d2e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b2477b63b6e44ae79f3ee69a08601912'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:54.234720 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1404/2819] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:31:54 2022] DELETE /identity/v3/users/b2477b63b6e44ae79f3ee69a08601912 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:31:54.509469 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14938b9e-2ff1-4554-8121-c85015dca6d0 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:54.510970 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-14938b9e-2ff1-4554-8121-c85015dca6d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:54.524142 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14938b9e-2ff1-4554-8121-c85015dca6d0 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:54.574604 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:54.583088 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a47ae1-bc88-47cd-9957-cce9d3f712dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:54.583729 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a47ae1-bc88-47cd-9957-cce9d3f712dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:54.584341 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a47ae1-bc88-47cd-9957-cce9d3f712dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:54.584662 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a47ae1-bc88-47cd-9957-cce9d3f712dc admin admin] PATH_INFO: `/v3/projects/59ce799c6ecc4e72819d87324d89a8c6` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:54.585795 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a47ae1-bc88-47cd-9957-cce9d3f712dc admin admin] RBAC: Authorizing `identity:delete_project(project_id=59ce799c6ecc4e72819d87324d89a8c6)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:54.588166 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a47ae1-bc88-47cd-9957-cce9d3f712dc admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:54.632638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-35a47ae1-bc88-47cd-9957-cce9d3f712dc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '59ce799c6ecc4e72819d87324d89a8c6', 'request_id': 'req-35a47ae1-bc88-47cd-9957-cce9d3f712dc'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:54.637999 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-35a47ae1-bc88-47cd-9957-cce9d3f712dc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '59ce799c6ecc4e72819d87324d89a8c6', 'request_id': 'req-35a47ae1-bc88-47cd-9957-cce9d3f712dc'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:54.644508 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-35a47ae1-bc88-47cd-9957-cce9d3f712dc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '59ce799c6ecc4e72819d87324d89a8c6', 'request_id': 'req-35a47ae1-bc88-47cd-9957-cce9d3f712dc'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:54.656719 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-35a47ae1-bc88-47cd-9957-cce9d3f712dc admin admin] The token cache is being invalidate because project 59ce799c6ecc4e72819d87324d89a8c6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:31:54.657002 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-35a47ae1-bc88-47cd-9957-cce9d3f712dc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:54.657626 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-35a47ae1-bc88-47cd-9957-cce9d3f712dc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:54.659318 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1416/2820] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:31:54 2022] DELETE /identity/v3/projects/59ce799c6ecc4e72819d87324d89a8c6 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:31:54.664144 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188a0132-e9e1-4214-845c-db6b6bc2e18e None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:54.664662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188a0132-e9e1-4214-845c-db6b6bc2e18e None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:54.664662 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188a0132-e9e1-4214-845c-db6b6bc2e18e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:54.683675 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-188a0132-e9e1-4214-845c-db6b6bc2e18e None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:31:54.684862 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-188a0132-e9e1-4214-845c-db6b6bc2e18e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:54.693539 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:54.749999 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1405/2821] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:31:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:31:54.757641 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e431b73-d176-4102-9446-cd15f13971d7 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:54.757962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e431b73-d176-4102-9446-cd15f13971d7 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:54.758258 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e431b73-d176-4102-9446-cd15f13971d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:54.778469 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-6e431b73-d176-4102-9446-cd15f13971d7 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:31:54.779662 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6e431b73-d176-4102-9446-cd15f13971d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:54.788513 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1417/2822] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:31:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:31:54.792650 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188a0132-e9e1-4214-845c-db6b6bc2e18e None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:54.802184 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188a0132-e9e1-4214-845c-db6b6bc2e18e None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:54.806640 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9dd149-b11d-4c2b-b302-c200547a3c71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:54.807224 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9dd149-b11d-4c2b-b302-c200547a3c71 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:54.807501 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9dd149-b11d-4c2b-b302-c200547a3c71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:54.807770 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9dd149-b11d-4c2b-b302-c200547a3c71 admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:54.808407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9dd149-b11d-4c2b-b302-c200547a3c71 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:31:54.808705 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9dd149-b11d-4c2b-b302-c200547a3c71 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:54.810301 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9dd149-b11d-4c2b-b302-c200547a3c71 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:54.821442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1406/2823] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:31:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:31:54.825592 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e431b73-d176-4102-9446-cd15f13971d7 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:54.835263 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e431b73-d176-4102-9446-cd15f13971d7 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:54.839862 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520bcaa3-cf8f-4c8e-af26-667a5a488da1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:54.840445 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520bcaa3-cf8f-4c8e-af26-667a5a488da1 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:54.840721 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520bcaa3-cf8f-4c8e-af26-667a5a488da1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:54.840997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520bcaa3-cf8f-4c8e-af26-667a5a488da1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:54.841615 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520bcaa3-cf8f-4c8e-af26-667a5a488da1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:54.843525 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520bcaa3-cf8f-4c8e-af26-667a5a488da1 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:54.870449 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1418/2824] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:31:54 2022] POST /identity/v3/projects => generated 371 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:31:54.874661 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9dd149-b11d-4c2b-b302-c200547a3c71 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:54.884286 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9dd149-b11d-4c2b-b302-c200547a3c71 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:54.939534 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f669e3-bc4c-4170-9f43-98fe575074c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:54.940122 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f669e3-bc4c-4170-9f43-98fe575074c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:54.940402 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f669e3-bc4c-4170-9f43-98fe575074c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:54.940668 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f669e3-bc4c-4170-9f43-98fe575074c6 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:54.941282 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f669e3-bc4c-4170-9f43-98fe575074c6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:54.943187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f669e3-bc4c-4170-9f43-98fe575074c6 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:54.963983 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1407/2825] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:31:54 2022] POST /identity/v3/users => generated 286 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:31:54.968241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520bcaa3-cf8f-4c8e-af26-667a5a488da1 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:54.978774 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520bcaa3-cf8f-4c8e-af26-667a5a488da1 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:54.983244 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950b168f-2dc7-4395-a785-0125abc95459 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:54.984077 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950b168f-2dc7-4395-a785-0125abc95459 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:54.985349 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950b168f-2dc7-4395-a785-0125abc95459 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:54.985349 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950b168f-2dc7-4395-a785-0125abc95459 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:54.985349 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-950b168f-2dc7-4395-a785-0125abc95459 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:54.986906 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-950b168f-2dc7-4395-a785-0125abc95459 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:54.996644 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1419/2826] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:31:54 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:31:55.000742 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f669e3-bc4c-4170-9f43-98fe575074c6 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:55.010564 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f669e3-bc4c-4170-9f43-98fe575074c6 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:55.015170 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b1a723-1e80-402a-b23f-e200ef708795 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:55.015756 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b1a723-1e80-402a-b23f-e200ef708795 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:55.016039 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b1a723-1e80-402a-b23f-e200ef708795 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:55.016312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b1a723-1e80-402a-b23f-e200ef708795 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:55.017446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b1a723-1e80-402a-b23f-e200ef708795 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:55.019052 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b1a723-1e80-402a-b23f-e200ef708795 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:55.029584 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1408/2827] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:31:54 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:31:55.034148 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950b168f-2dc7-4395-a785-0125abc95459 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:55.044354 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950b168f-2dc7-4395-a785-0125abc95459 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:55.048985 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8277c1-44c1-4eb1-b079-1be35fcce2d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:55.049619 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8277c1-44c1-4eb1-b079-1be35fcce2d8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:55.049935 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8277c1-44c1-4eb1-b079-1be35fcce2d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:55.050211 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8277c1-44c1-4eb1-b079-1be35fcce2d8 admin admin] PATH_INFO: `/v3/projects/db1eef069b6f4ea1bbed2547506998c4/users/719ccf82992440b49827351da0feffba/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:55.065560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8277c1-44c1-4eb1-b079-1be35fcce2d8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=db1eef069b6f4ea1bbed2547506998c4, user_id=719ccf82992440b49827351da0feffba, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:55.068222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8277c1-44c1-4eb1-b079-1be35fcce2d8 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:55.083386 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1420/2828] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:31:55 2022] PUT /identity/v3/projects/db1eef069b6f4ea1bbed2547506998c4/users/719ccf82992440b49827351da0feffba/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:31:55.095726 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d93b590-63f4-4baa-8a78-a7c83f436d1a None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:55.096010 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d93b590-63f4-4baa-8a78-a7c83f436d1a None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:55.096310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d93b590-63f4-4baa-8a78-a7c83f436d1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:55.126719 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-0d93b590-63f4-4baa-8a78-a7c83f436d1a None None] MFA Rules not processed for user `719ccf82992440b49827351da0feffba`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:31:55.127913 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0d93b590-63f4-4baa-8a78-a7c83f436d1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:55.191018 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1409/2829] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:31:55 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:31:55.202480 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8277c1-44c1-4eb1-b079-1be35fcce2d8 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:55.204734 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-3c8277c1-44c1-4eb1-b079-1be35fcce2d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:55.260387 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8277c1-44c1-4eb1-b079-1be35fcce2d8 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:55.263908 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1288c104-0889-4c92-8d05-9d427ba411a2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:55.264503 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1288c104-0889-4c92-8d05-9d427ba411a2 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:55.264775 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1288c104-0889-4c92-8d05-9d427ba411a2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:55.265042 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1288c104-0889-4c92-8d05-9d427ba411a2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:55.273563 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1288c104-0889-4c92-8d05-9d427ba411a2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:55.274999 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1288c104-0889-4c92-8d05-9d427ba411a2 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:55.283698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1421/2830] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:31:55 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:31:55.305894 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d93b590-63f4-4baa-8a78-a7c83f436d1a None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:55.316103 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:55.369312 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d93b590-63f4-4baa-8a78-a7c83f436d1a None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:55.374080 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3440a1-3e30-4882-8a41-1b2fef2dbe09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:55.374697 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3440a1-3e30-4882-8a41-1b2fef2dbe09 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:55.374977 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3440a1-3e30-4882-8a41-1b2fef2dbe09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:55.375249 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3440a1-3e30-4882-8a41-1b2fef2dbe09 admin admin] PATH_INFO: `/v3/users/719ccf82992440b49827351da0feffba` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:55.376387 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb3440a1-3e30-4882-8a41-1b2fef2dbe09 admin admin] RBAC: Authorizing `identity:delete_user(user_id=719ccf82992440b49827351da0feffba)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:55.378313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb3440a1-3e30-4882-8a41-1b2fef2dbe09 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:55.428363 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-bb3440a1-3e30-4882-8a41-1b2fef2dbe09 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '719ccf82992440b49827351da0feffba'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:55.435183 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-bb3440a1-3e30-4882-8a41-1b2fef2dbe09 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '719ccf82992440b49827351da0feffba'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:55.435876 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-bb3440a1-3e30-4882-8a41-1b2fef2dbe09 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '719ccf82992440b49827351da0feffba'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:55.443081 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-bb3440a1-3e30-4882-8a41-1b2fef2dbe09 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '719ccf82992440b49827351da0feffba'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:55.454795 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-bb3440a1-3e30-4882-8a41-1b2fef2dbe09 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '719ccf82992440b49827351da0feffba'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:55.460941 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1410/2831] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:31:55 2022] DELETE /identity/v3/users/719ccf82992440b49827351da0feffba => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:31:55.465435 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a505226c-3798-4c0f-be92-4069e16f2382 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:55.465712 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a505226c-3798-4c0f-be92-4069e16f2382 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:55.466018 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a505226c-3798-4c0f-be92-4069e16f2382 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:55.485420 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-a505226c-3798-4c0f-be92-4069e16f2382 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:31:55.486638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a505226c-3798-4c0f-be92-4069e16f2382 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:55.505521 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:55.544923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1422/2832] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:31:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:31:55.555137 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3440a1-3e30-4882-8a41-1b2fef2dbe09 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:55.556497 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-bb3440a1-3e30-4882-8a41-1b2fef2dbe09 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:55.614865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3440a1-3e30-4882-8a41-1b2fef2dbe09 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:55.618516 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0ffb1a-d0ca-489b-a8ce-5a17a59b614b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:55.619114 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0ffb1a-d0ca-489b-a8ce-5a17a59b614b service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:55.619395 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0ffb1a-d0ca-489b-a8ce-5a17a59b614b service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:55.619672 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0ffb1a-d0ca-489b-a8ce-5a17a59b614b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:55.629120 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0ffb1a-d0ca-489b-a8ce-5a17a59b614b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:55.630581 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0ffb1a-d0ca-489b-a8ce-5a17a59b614b service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:55.639908 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1411/2833] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:31:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:31:55.900967 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a505226c-3798-4c0f-be92-4069e16f2382 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:55.902420 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a505226c-3798-4c0f-be92-4069e16f2382 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:55.916078 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a505226c-3798-4c0f-be92-4069e16f2382 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:55.920689 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a7fe93-a551-43cf-a22f-d5e58b7d861e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:55.921321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a7fe93-a551-43cf-a22f-d5e58b7d861e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:55.921597 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a7fe93-a551-43cf-a22f-d5e58b7d861e admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:55.921907 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a7fe93-a551-43cf-a22f-d5e58b7d861e admin admin] PATH_INFO: `/v3/projects/db1eef069b6f4ea1bbed2547506998c4` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:55.924107 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a7fe93-a551-43cf-a22f-d5e58b7d861e admin admin] RBAC: Authorizing `identity:delete_project(project_id=db1eef069b6f4ea1bbed2547506998c4)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:55.926264 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a7fe93-a551-43cf-a22f-d5e58b7d861e admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:55.972807 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-72a7fe93-a551-43cf-a22f-d5e58b7d861e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'db1eef069b6f4ea1bbed2547506998c4', 'request_id': 'req-72a7fe93-a551-43cf-a22f-d5e58b7d861e'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:55.978406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-72a7fe93-a551-43cf-a22f-d5e58b7d861e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'db1eef069b6f4ea1bbed2547506998c4', 'request_id': 'req-72a7fe93-a551-43cf-a22f-d5e58b7d861e'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:55.984438 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-72a7fe93-a551-43cf-a22f-d5e58b7d861e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'db1eef069b6f4ea1bbed2547506998c4', 'request_id': 'req-72a7fe93-a551-43cf-a22f-d5e58b7d861e'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:55.996999 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-72a7fe93-a551-43cf-a22f-d5e58b7d861e admin admin] The token cache is being invalidate because project db1eef069b6f4ea1bbed2547506998c4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:31:55.997281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-72a7fe93-a551-43cf-a22f-d5e58b7d861e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:55.997927 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-72a7fe93-a551-43cf-a22f-d5e58b7d861e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:55.999403 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1423/2834] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:31:55 2022] DELETE /identity/v3/projects/db1eef069b6f4ea1bbed2547506998c4 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:31:56.004409 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1af08f-25a6-44c0-9b7b-8466d9af3eef None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:56.004881 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1af08f-25a6-44c0-9b7b-8466d9af3eef None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:56.004980 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1af08f-25a6-44c0-9b7b-8466d9af3eef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:56.024404 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-0a1af08f-25a6-44c0-9b7b-8466d9af3eef None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:31:56.025909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0a1af08f-25a6-44c0-9b7b-8466d9af3eef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:56.052048 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:56.087004 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1412/2835] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:31:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:31:56.095228 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc6ad14-a0fa-403a-b1d1-fe3e598c99bf None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:56.095507 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc6ad14-a0fa-403a-b1d1-fe3e598c99bf None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:56.095782 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc6ad14-a0fa-403a-b1d1-fe3e598c99bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:56.115490 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-9bc6ad14-a0fa-403a-b1d1-fe3e598c99bf None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:31:56.116689 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-9bc6ad14-a0fa-403a-b1d1-fe3e598c99bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:56.124514 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1424/2836] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:31:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:31:56.128996 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1af08f-25a6-44c0-9b7b-8466d9af3eef None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:56.138958 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1af08f-25a6-44c0-9b7b-8466d9af3eef None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:56.143646 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932f70bb-f079-4975-9e81-73d0cdb96f54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:56.144235 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932f70bb-f079-4975-9e81-73d0cdb96f54 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:56.144515 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932f70bb-f079-4975-9e81-73d0cdb96f54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:56.144790 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932f70bb-f079-4975-9e81-73d0cdb96f54 admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:56.145428 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932f70bb-f079-4975-9e81-73d0cdb96f54 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:31:56.145728 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932f70bb-f079-4975-9e81-73d0cdb96f54 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:56.147331 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932f70bb-f079-4975-9e81-73d0cdb96f54 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:56.159556 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1413/2837] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:31:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:31:56.163850 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc6ad14-a0fa-403a-b1d1-fe3e598c99bf None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:56.173660 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc6ad14-a0fa-403a-b1d1-fe3e598c99bf None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:56.178304 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91c5710-de78-4f35-ac12-97fea4127868 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:56.178894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91c5710-de78-4f35-ac12-97fea4127868 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:56.179171 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91c5710-de78-4f35-ac12-97fea4127868 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:56.179443 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91c5710-de78-4f35-ac12-97fea4127868 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:56.180064 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91c5710-de78-4f35-ac12-97fea4127868 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:56.181961 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91c5710-de78-4f35-ac12-97fea4127868 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:56.209091 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1425/2838] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:31:56 2022] POST /identity/v3/projects => generated 375 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:31:56.213160 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932f70bb-f079-4975-9e81-73d0cdb96f54 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:56.223502 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932f70bb-f079-4975-9e81-73d0cdb96f54 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:56.279909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92a66ab-95e9-4f2c-b352-05cb42a00f02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:56.280505 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92a66ab-95e9-4f2c-b352-05cb42a00f02 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:56.280787 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92a66ab-95e9-4f2c-b352-05cb42a00f02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:56.281059 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92a66ab-95e9-4f2c-b352-05cb42a00f02 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:56.281688 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92a66ab-95e9-4f2c-b352-05cb42a00f02 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:56.283658 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92a66ab-95e9-4f2c-b352-05cb42a00f02 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:56.304345 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1414/2839] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:31:56 2022] POST /identity/v3/users => generated 288 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:31:56.308366 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91c5710-de78-4f35-ac12-97fea4127868 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:56.318851 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91c5710-de78-4f35-ac12-97fea4127868 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:56.323322 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d53034-2b2d-4fc0-ab66-eaa296ea76f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:56.323902 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d53034-2b2d-4fc0-ab66-eaa296ea76f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:56.324174 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d53034-2b2d-4fc0-ab66-eaa296ea76f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:56.324442 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d53034-2b2d-4fc0-ab66-eaa296ea76f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:56.325072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d53034-2b2d-4fc0-ab66-eaa296ea76f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:56.326679 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d53034-2b2d-4fc0-ab66-eaa296ea76f9 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:56.337321 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1426/2840] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:31:56 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:31:56.340968 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92a66ab-95e9-4f2c-b352-05cb42a00f02 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:56.350595 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92a66ab-95e9-4f2c-b352-05cb42a00f02 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:56.355064 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb28fe6a-0206-4a1d-8a50-78187a4e986b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:56.355637 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb28fe6a-0206-4a1d-8a50-78187a4e986b admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:56.355914 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb28fe6a-0206-4a1d-8a50-78187a4e986b admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:56.356180 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb28fe6a-0206-4a1d-8a50-78187a4e986b admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:56.356804 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb28fe6a-0206-4a1d-8a50-78187a4e986b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:56.358380 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb28fe6a-0206-4a1d-8a50-78187a4e986b admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:56.369067 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1415/2841] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:31:56 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:31:56.372932 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d53034-2b2d-4fc0-ab66-eaa296ea76f9 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:56.383397 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d53034-2b2d-4fc0-ab66-eaa296ea76f9 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:56.387925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f10f83-0e44-410b-bcd8-35dd00316c8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:56.388560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f10f83-0e44-410b-bcd8-35dd00316c8d admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:56.388836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f10f83-0e44-410b-bcd8-35dd00316c8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:56.389111 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f10f83-0e44-410b-bcd8-35dd00316c8d admin admin] PATH_INFO: `/v3/projects/3e170049f0bc4a6ba4c4b513591e053c/users/8c8728c5d1864f9faea044e0df51afe5/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:56.403588 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f10f83-0e44-410b-bcd8-35dd00316c8d admin admin] RBAC: Authorizing `identity:create_grant(project_id=3e170049f0bc4a6ba4c4b513591e053c, user_id=8c8728c5d1864f9faea044e0df51afe5, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:56.406241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f10f83-0e44-410b-bcd8-35dd00316c8d admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:56.418927 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1427/2842] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:31:56 2022] PUT /identity/v3/projects/3e170049f0bc4a6ba4c4b513591e053c/users/8c8728c5d1864f9faea044e0df51afe5/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:31:56.430684 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f0acfa-a4fc-4e6e-88d7-f2764afbcbbb None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:56.430977 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f0acfa-a4fc-4e6e-88d7-f2764afbcbbb None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:56.431272 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f0acfa-a4fc-4e6e-88d7-f2764afbcbbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:56.458828 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-f3f0acfa-a4fc-4e6e-88d7-f2764afbcbbb None None] MFA Rules not processed for user `8c8728c5d1864f9faea044e0df51afe5`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:31:56.460032 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-f3f0acfa-a4fc-4e6e-88d7-f2764afbcbbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:56.522357 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1416/2843] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:31:56 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:31:56.527045 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f10f83-0e44-410b-bcd8-35dd00316c8d admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:56.573922 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:56.590015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f10f83-0e44-410b-bcd8-35dd00316c8d admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:56.594576 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75b1941-42f8-4e64-b70d-0f24dbc1bb4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:56.595171 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75b1941-42f8-4e64-b70d-0f24dbc1bb4d admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:56.595451 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75b1941-42f8-4e64-b70d-0f24dbc1bb4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:56.595727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75b1941-42f8-4e64-b70d-0f24dbc1bb4d admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:56.596356 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a75b1941-42f8-4e64-b70d-0f24dbc1bb4d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:56.598269 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a75b1941-42f8-4e64-b70d-0f24dbc1bb4d admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:56.624237 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1428/2844] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:31:56 2022] POST /identity/v3/projects => generated 375 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:31:56.628895 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f0acfa-a4fc-4e6e-88d7-f2764afbcbbb None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:56.639530 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f0acfa-a4fc-4e6e-88d7-f2764afbcbbb None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:56.663843 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:56.697767 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3fc62dc-d505-4586-8c6c-a917a7c3191d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:56.698373 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fc62dc-d505-4586-8c6c-a917a7c3191d admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:56.698649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fc62dc-d505-4586-8c6c-a917a7c3191d admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:56.698919 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fc62dc-d505-4586-8c6c-a917a7c3191d admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:56.699542 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3fc62dc-d505-4586-8c6c-a917a7c3191d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:56.701429 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3fc62dc-d505-4586-8c6c-a917a7c3191d admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:56.721708 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1417/2845] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:31:56 2022] POST /identity/v3/users => generated 294 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:31:56.725848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75b1941-42f8-4e64-b70d-0f24dbc1bb4d admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:56.736570 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75b1941-42f8-4e64-b70d-0f24dbc1bb4d admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:56.741144 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d52d21-876d-442a-af8d-4bcadd8cd4c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:56.741731 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d52d21-876d-442a-af8d-4bcadd8cd4c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:56.742040 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d52d21-876d-442a-af8d-4bcadd8cd4c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:56.742311 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d52d21-876d-442a-af8d-4bcadd8cd4c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:56.742947 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d52d21-876d-442a-af8d-4bcadd8cd4c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:56.744525 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d52d21-876d-442a-af8d-4bcadd8cd4c8 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:56.754993 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1429/2846] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:31:56 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:31:56.758897 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3fc62dc-d505-4586-8c6c-a917a7c3191d admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:56.768704 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3fc62dc-d505-4586-8c6c-a917a7c3191d admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:56.773219 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f2d302-ce51-4d46-b332-4cd5215b3e71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:56.773881 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f2d302-ce51-4d46-b332-4cd5215b3e71 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:56.774161 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f2d302-ce51-4d46-b332-4cd5215b3e71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:56.774428 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f2d302-ce51-4d46-b332-4cd5215b3e71 admin admin] PATH_INFO: `/v3/projects/27ffb402f2904efe87077e4d1ff8c091/users/3aec07d40e9e4b1da8092c9fe587294b/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:56.790610 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f2d302-ce51-4d46-b332-4cd5215b3e71 admin admin] RBAC: Authorizing `identity:create_grant(project_id=27ffb402f2904efe87077e4d1ff8c091, user_id=3aec07d40e9e4b1da8092c9fe587294b, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:56.793173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f2d302-ce51-4d46-b332-4cd5215b3e71 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:56.806152 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1418/2847] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:31:56 2022] PUT /identity/v3/projects/27ffb402f2904efe87077e4d1ff8c091/users/3aec07d40e9e4b1da8092c9fe587294b/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:31:56.810055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d52d21-876d-442a-af8d-4bcadd8cd4c8 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:56.819849 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d52d21-876d-442a-af8d-4bcadd8cd4c8 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:56.874445 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8957a312-3d53-4a6a-b0f0-ea5b1b518577 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:56.874976 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8957a312-3d53-4a6a-b0f0-ea5b1b518577 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:56.875255 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8957a312-3d53-4a6a-b0f0-ea5b1b518577 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:56.875555 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8957a312-3d53-4a6a-b0f0-ea5b1b518577 admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:56.876187 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8957a312-3d53-4a6a-b0f0-ea5b1b518577 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:56.878358 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8957a312-3d53-4a6a-b0f0-ea5b1b518577 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:56.887913 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1430/2848] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:31:56 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:31:56.892063 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f2d302-ce51-4d46-b332-4cd5215b3e71 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:56.902443 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f2d302-ce51-4d46-b332-4cd5215b3e71 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:56.907454 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714f5913-610f-4543-a78f-ad650d47895a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:56.908084 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714f5913-610f-4543-a78f-ad650d47895a admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:56.908363 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714f5913-610f-4543-a78f-ad650d47895a admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:56.908635 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714f5913-610f-4543-a78f-ad650d47895a admin admin] PATH_INFO: `/v3/projects/27ffb402f2904efe87077e4d1ff8c091/users/3aec07d40e9e4b1da8092c9fe587294b/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:56.910604 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714f5913-610f-4543-a78f-ad650d47895a admin admin] RBAC: Authorizing `identity:create_grant(project_id=27ffb402f2904efe87077e4d1ff8c091, user_id=3aec07d40e9e4b1da8092c9fe587294b, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:56.913155 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714f5913-610f-4543-a78f-ad650d47895a admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:56.923588 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1419/2849] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:31:56 2022] PUT /identity/v3/projects/27ffb402f2904efe87077e4d1ff8c091/users/3aec07d40e9e4b1da8092c9fe587294b/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:31:56.933557 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2ff090-454a-4606-8c13-82441b217a7b None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:56.933921 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2ff090-454a-4606-8c13-82441b217a7b None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:56.934200 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2ff090-454a-4606-8c13-82441b217a7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:56.960815 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-9b2ff090-454a-4606-8c13-82441b217a7b None None] MFA Rules not processed for user `3aec07d40e9e4b1da8092c9fe587294b`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:31:56.962076 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-9b2ff090-454a-4606-8c13-82441b217a7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:57.028423 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1431/2850] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:31:56 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:31:57.039768 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714f5913-610f-4543-a78f-ad650d47895a admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:57.041483 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-714f5913-610f-4543-a78f-ad650d47895a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:57.096791 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714f5913-610f-4543-a78f-ad650d47895a admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:57.100337 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5b35e7-472c-4569-92f0-4189db381d0f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:57.100917 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5b35e7-472c-4569-92f0-4189db381d0f service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:57.101221 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5b35e7-472c-4569-92f0-4189db381d0f service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:57.101491 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5b35e7-472c-4569-92f0-4189db381d0f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:57.161541 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac5b35e7-472c-4569-92f0-4189db381d0f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:57.163667 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac5b35e7-472c-4569-92f0-4189db381d0f service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:57.172840 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1420/2851] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:31:57 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:31:59.696898 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2ff090-454a-4606-8c13-82441b217a7b None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:59.740980 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:59.759540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2ff090-454a-4606-8c13-82441b217a7b None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:31:59.763991 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d7a24a-25ce-41dd-8332-6ebb789af1ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:31:59.764613 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d7a24a-25ce-41dd-8332-6ebb789af1ae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:59.764892 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d7a24a-25ce-41dd-8332-6ebb789af1ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:59.765164 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d7a24a-25ce-41dd-8332-6ebb789af1ae admin admin] PATH_INFO: `/v3/users/8c8728c5d1864f9faea044e0df51afe5` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:59.766299 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d7a24a-25ce-41dd-8332-6ebb789af1ae admin admin] RBAC: Authorizing `identity:delete_user(user_id=8c8728c5d1864f9faea044e0df51afe5)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:31:59.768215 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d7a24a-25ce-41dd-8332-6ebb789af1ae admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:31:59.813328 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a6d7a24a-25ce-41dd-8332-6ebb789af1ae admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8c8728c5d1864f9faea044e0df51afe5'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:59.820091 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a6d7a24a-25ce-41dd-8332-6ebb789af1ae admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8c8728c5d1864f9faea044e0df51afe5'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:59.820719 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a6d7a24a-25ce-41dd-8332-6ebb789af1ae admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8c8728c5d1864f9faea044e0df51afe5'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:59.827186 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a6d7a24a-25ce-41dd-8332-6ebb789af1ae admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8c8728c5d1864f9faea044e0df51afe5'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:59.839059 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-a6d7a24a-25ce-41dd-8332-6ebb789af1ae admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8c8728c5d1864f9faea044e0df51afe5'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:31:59.844798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1432/2852] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:31:59 2022] DELETE /identity/v3/users/8c8728c5d1864f9faea044e0df51afe5 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:31:59.849353 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf6d97b-4a58-4aaa-82c5-bbccff78e526 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:31:59.849865 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf6d97b-4a58-4aaa-82c5-bbccff78e526 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:31:59.849967 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf6d97b-4a58-4aaa-82c5-bbccff78e526 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:31:59.868762 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-0bf6d97b-4a58-4aaa-82c5-bbccff78e526 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:31:59.869992 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0bf6d97b-4a58-4aaa-82c5-bbccff78e526 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:59.898972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:31:59.929869 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1421/2853] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:31:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:31:59.940195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d7a24a-25ce-41dd-8332-6ebb789af1ae admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:31:59.941567 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a6d7a24a-25ce-41dd-8332-6ebb789af1ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:31:59.996705 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d7a24a-25ce-41dd-8332-6ebb789af1ae admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:00.000331 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09213a0a-b870-4b60-a37e-92237d235f40 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:00.000921 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09213a0a-b870-4b60-a37e-92237d235f40 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:00.001195 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09213a0a-b870-4b60-a37e-92237d235f40 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:00.001466 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09213a0a-b870-4b60-a37e-92237d235f40 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:00.010214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09213a0a-b870-4b60-a37e-92237d235f40 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:00.011622 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09213a0a-b870-4b60-a37e-92237d235f40 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:00.021252 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1433/2854] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:31:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:32:00.286633 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf6d97b-4a58-4aaa-82c5-bbccff78e526 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:00.288163 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-0bf6d97b-4a58-4aaa-82c5-bbccff78e526 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:00.300274 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf6d97b-4a58-4aaa-82c5-bbccff78e526 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:00.304896 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af667953-0ab3-40e8-afa0-a58419240d62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:00.305584 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af667953-0ab3-40e8-afa0-a58419240d62 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:00.305837 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af667953-0ab3-40e8-afa0-a58419240d62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:00.306113 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af667953-0ab3-40e8-afa0-a58419240d62 admin admin] PATH_INFO: `/v3/projects/3e170049f0bc4a6ba4c4b513591e053c` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:00.307278 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af667953-0ab3-40e8-afa0-a58419240d62 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3e170049f0bc4a6ba4c4b513591e053c)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:00.309146 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af667953-0ab3-40e8-afa0-a58419240d62 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:00.356097 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-af667953-0ab3-40e8-afa0-a58419240d62 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3e170049f0bc4a6ba4c4b513591e053c', 'request_id': 'req-af667953-0ab3-40e8-afa0-a58419240d62'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:00.361574 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-af667953-0ab3-40e8-afa0-a58419240d62 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3e170049f0bc4a6ba4c4b513591e053c', 'request_id': 'req-af667953-0ab3-40e8-afa0-a58419240d62'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:00.367575 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-af667953-0ab3-40e8-afa0-a58419240d62 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3e170049f0bc4a6ba4c4b513591e053c', 'request_id': 'req-af667953-0ab3-40e8-afa0-a58419240d62'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:00.380106 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-af667953-0ab3-40e8-afa0-a58419240d62 admin admin] The token cache is being invalidate because project 3e170049f0bc4a6ba4c4b513591e053c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:32:00.380394 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-af667953-0ab3-40e8-afa0-a58419240d62 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:00.381015 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-af667953-0ab3-40e8-afa0-a58419240d62 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:00.382526 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1422/2855] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:32:00 2022] DELETE /identity/v3/projects/3e170049f0bc4a6ba4c4b513591e053c => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:32:00.386641 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09213a0a-b870-4b60-a37e-92237d235f40 service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:00.396549 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09213a0a-b870-4b60-a37e-92237d235f40 service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:00.452628 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd06ebf6-6c27-4328-949b-010bd081b5e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:00.453250 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd06ebf6-6c27-4328-949b-010bd081b5e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:00.453526 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd06ebf6-6c27-4328-949b-010bd081b5e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:00.453941 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd06ebf6-6c27-4328-949b-010bd081b5e9 admin admin] PATH_INFO: `/v3/users/3aec07d40e9e4b1da8092c9fe587294b` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:00.455092 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd06ebf6-6c27-4328-949b-010bd081b5e9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3aec07d40e9e4b1da8092c9fe587294b)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:00.457011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd06ebf6-6c27-4328-949b-010bd081b5e9 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:00.461378 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:32:00.503659 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-bd06ebf6-6c27-4328-949b-010bd081b5e9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3aec07d40e9e4b1da8092c9fe587294b'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:00.510374 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-bd06ebf6-6c27-4328-949b-010bd081b5e9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3aec07d40e9e4b1da8092c9fe587294b'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:00.511015 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-bd06ebf6-6c27-4328-949b-010bd081b5e9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3aec07d40e9e4b1da8092c9fe587294b'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:00.518192 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-bd06ebf6-6c27-4328-949b-010bd081b5e9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3aec07d40e9e4b1da8092c9fe587294b'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:00.529984 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-bd06ebf6-6c27-4328-949b-010bd081b5e9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3aec07d40e9e4b1da8092c9fe587294b'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:00.535617 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1434/2856] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:32:00 2022] DELETE /identity/v3/users/3aec07d40e9e4b1da8092c9fe587294b => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:32:00.798452 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af667953-0ab3-40e8-afa0-a58419240d62 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:00.799870 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-af667953-0ab3-40e8-afa0-a58419240d62 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:00.813389 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af667953-0ab3-40e8-afa0-a58419240d62 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:00.868163 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53814595-c6c1-4fbe-81fb-e4ace377cbae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:00.868833 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53814595-c6c1-4fbe-81fb-e4ace377cbae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:00.869053 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53814595-c6c1-4fbe-81fb-e4ace377cbae admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:00.869321 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53814595-c6c1-4fbe-81fb-e4ace377cbae admin admin] PATH_INFO: `/v3/projects/27ffb402f2904efe87077e4d1ff8c091` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:00.870428 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53814595-c6c1-4fbe-81fb-e4ace377cbae admin admin] RBAC: Authorizing `identity:delete_project(project_id=27ffb402f2904efe87077e4d1ff8c091)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:00.872415 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53814595-c6c1-4fbe-81fb-e4ace377cbae admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:00.917702 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-53814595-c6c1-4fbe-81fb-e4ace377cbae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '27ffb402f2904efe87077e4d1ff8c091', 'request_id': 'req-53814595-c6c1-4fbe-81fb-e4ace377cbae'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:00.923115 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-53814595-c6c1-4fbe-81fb-e4ace377cbae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '27ffb402f2904efe87077e4d1ff8c091', 'request_id': 'req-53814595-c6c1-4fbe-81fb-e4ace377cbae'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:00.924169 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:32:00.930677 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-53814595-c6c1-4fbe-81fb-e4ace377cbae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '27ffb402f2904efe87077e4d1ff8c091', 'request_id': 'req-53814595-c6c1-4fbe-81fb-e4ace377cbae'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:00.943191 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-53814595-c6c1-4fbe-81fb-e4ace377cbae admin admin] The token cache is being invalidate because project 27ffb402f2904efe87077e4d1ff8c091 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:32:00.943481 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-53814595-c6c1-4fbe-81fb-e4ace377cbae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:00.944094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-53814595-c6c1-4fbe-81fb-e4ace377cbae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:00.945543 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1423/2857] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:32:00 2022] DELETE /identity/v3/projects/27ffb402f2904efe87077e4d1ff8c091 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:32:00.950755 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17255860-cd09-430e-8f6a-02ef771ff32d None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:00.951217 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17255860-cd09-430e-8f6a-02ef771ff32d None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:00.951338 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17255860-cd09-430e-8f6a-02ef771ff32d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:00.970784 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-17255860-cd09-430e-8f6a-02ef771ff32d None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:32:00.972011 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-17255860-cd09-430e-8f6a-02ef771ff32d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:01.030191 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1435/2858] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:32:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:32:01.037818 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb798c23-222a-4af5-9668-c9b1cd61fbb4 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:01.038110 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb798c23-222a-4af5-9668-c9b1cd61fbb4 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:01.038387 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb798c23-222a-4af5-9668-c9b1cd61fbb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:01.057486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-fb798c23-222a-4af5-9668-c9b1cd61fbb4 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:32:01.058652 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-fb798c23-222a-4af5-9668-c9b1cd61fbb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:01.066965 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1424/2859] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:32:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:32:01.071061 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17255860-cd09-430e-8f6a-02ef771ff32d None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:01.081257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17255860-cd09-430e-8f6a-02ef771ff32d None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:01.085698 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930a274e-6f83-4e4c-be96-3f0ea1512223 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:01.086310 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930a274e-6f83-4e4c-be96-3f0ea1512223 admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:01.086581 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930a274e-6f83-4e4c-be96-3f0ea1512223 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:01.086852 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930a274e-6f83-4e4c-be96-3f0ea1512223 admin admin] PATH_INFO: `/v3/domains` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:01.087483 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-930a274e-6f83-4e4c-be96-3f0ea1512223 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81420) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:32:01.087775 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-930a274e-6f83-4e4c-be96-3f0ea1512223 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:01.089342 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-930a274e-6f83-4e4c-be96-3f0ea1512223 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:01.100438 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1436/2860] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:32:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:32:01.104521 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb798c23-222a-4af5-9668-c9b1cd61fbb4 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:01.113568 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb798c23-222a-4af5-9668-c9b1cd61fbb4 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:01.118302 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45834d4b-732d-437e-a4cf-30558d3e2135 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:01.118888 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45834d4b-732d-437e-a4cf-30558d3e2135 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:01.119165 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45834d4b-732d-437e-a4cf-30558d3e2135 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:01.119442 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45834d4b-732d-437e-a4cf-30558d3e2135 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:01.120062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45834d4b-732d-437e-a4cf-30558d3e2135 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:01.121986 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45834d4b-732d-437e-a4cf-30558d3e2135 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:01.148490 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1425/2861] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:32:01 2022] POST /identity/v3/projects => generated 377 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:32:01.152757 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930a274e-6f83-4e4c-be96-3f0ea1512223 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:01.162962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930a274e-6f83-4e4c-be96-3f0ea1512223 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:01.217978 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7330392b-057a-43ae-a488-242c360f1f49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:01.218628 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7330392b-057a-43ae-a488-242c360f1f49 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:01.218925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7330392b-057a-43ae-a488-242c360f1f49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:01.219182 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7330392b-057a-43ae-a488-242c360f1f49 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:01.219801 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7330392b-057a-43ae-a488-242c360f1f49 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:01.221694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7330392b-057a-43ae-a488-242c360f1f49 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:01.238141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:32:01.246495 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1437/2862] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:32:01 2022] POST /identity/v3/users => generated 289 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:32:01.247678 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45834d4b-732d-437e-a4cf-30558d3e2135 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:01.258503 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45834d4b-732d-437e-a4cf-30558d3e2135 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:01.263086 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1361b523-8ea8-4ffa-84b4-14634662254f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:01.263672 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1361b523-8ea8-4ffa-84b4-14634662254f admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:01.263951 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1361b523-8ea8-4ffa-84b4-14634662254f admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:01.264221 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1361b523-8ea8-4ffa-84b4-14634662254f admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:01.264854 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1361b523-8ea8-4ffa-84b4-14634662254f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:01.266473 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1361b523-8ea8-4ffa-84b4-14634662254f admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:01.276786 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1426/2863] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:32:01 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:32:01.280598 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7330392b-057a-43ae-a488-242c360f1f49 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:01.290414 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7330392b-057a-43ae-a488-242c360f1f49 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:01.295277 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2f58b7-72ab-45d7-9af0-3d4a26fc49cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:01.295874 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2f58b7-72ab-45d7-9af0-3d4a26fc49cd admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:01.296151 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2f58b7-72ab-45d7-9af0-3d4a26fc49cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:01.296421 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2f58b7-72ab-45d7-9af0-3d4a26fc49cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:01.297605 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2f58b7-72ab-45d7-9af0-3d4a26fc49cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:01.299224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2f58b7-72ab-45d7-9af0-3d4a26fc49cd admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:01.310055 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1438/2864] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:32:01 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:32:01.314653 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1361b523-8ea8-4ffa-84b4-14634662254f admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:01.324430 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1361b523-8ea8-4ffa-84b4-14634662254f admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:01.328915 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7476de00-1c01-48cb-a572-32c62c9d0d21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:01.329544 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7476de00-1c01-48cb-a572-32c62c9d0d21 admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:01.329854 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7476de00-1c01-48cb-a572-32c62c9d0d21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:01.330154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7476de00-1c01-48cb-a572-32c62c9d0d21 admin admin] PATH_INFO: `/v3/projects/710392938fb8429b991aab0d48faf1d7/users/8859df7b5d7f45c6bb2fad6896c87539/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:01.344204 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7476de00-1c01-48cb-a572-32c62c9d0d21 admin admin] RBAC: Authorizing `identity:create_grant(project_id=710392938fb8429b991aab0d48faf1d7, user_id=8859df7b5d7f45c6bb2fad6896c87539, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:01.346835 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7476de00-1c01-48cb-a572-32c62c9d0d21 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:01.360133 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1427/2865] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:32:01 2022] PUT /identity/v3/projects/710392938fb8429b991aab0d48faf1d7/users/8859df7b5d7f45c6bb2fad6896c87539/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:32:01.371328 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7c22cf-179a-4363-8ab0-73ad48519343 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:01.371609 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7c22cf-179a-4363-8ab0-73ad48519343 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:01.371878 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7c22cf-179a-4363-8ab0-73ad48519343 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:01.399211 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-4f7c22cf-179a-4363-8ab0-73ad48519343 None None] MFA Rules not processed for user `8859df7b5d7f45c6bb2fad6896c87539`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:32:01.400659 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-4f7c22cf-179a-4363-8ab0-73ad48519343 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:01.462344 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1439/2866] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:32:01 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:32:01.466802 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7476de00-1c01-48cb-a572-32c62c9d0d21 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:01.527359 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7476de00-1c01-48cb-a572-32c62c9d0d21 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:01.532463 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e057f2d7-7626-45ad-84ff-49121d958fc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:01.533055 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e057f2d7-7626-45ad-84ff-49121d958fc6 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:01.533332 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e057f2d7-7626-45ad-84ff-49121d958fc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:01.533603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e057f2d7-7626-45ad-84ff-49121d958fc6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:01.534255 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e057f2d7-7626-45ad-84ff-49121d958fc6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:01.536113 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e057f2d7-7626-45ad-84ff-49121d958fc6 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:01.563029 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1428/2867] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:32:01 2022] POST /identity/v3/projects => generated 377 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:32:01.566960 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7c22cf-179a-4363-8ab0-73ad48519343 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:01.577640 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7c22cf-179a-4363-8ab0-73ad48519343 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:01.632511 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd2f118-48c6-48d8-8941-06b51f6bf8c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:01.633099 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd2f118-48c6-48d8-8941-06b51f6bf8c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:01.633372 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd2f118-48c6-48d8-8941-06b51f6bf8c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:01.633639 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd2f118-48c6-48d8-8941-06b51f6bf8c3 admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:01.634284 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd2f118-48c6-48d8-8941-06b51f6bf8c3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:01.636162 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd2f118-48c6-48d8-8941-06b51f6bf8c3 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:01.657540 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1440/2868] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:32:01 2022] POST /identity/v3/users => generated 289 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:32:01.661441 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e057f2d7-7626-45ad-84ff-49121d958fc6 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:01.671528 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e057f2d7-7626-45ad-84ff-49121d958fc6 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:01.676279 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3785a9-0b0e-47e8-92e0-dd2a92a46372 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:01.676894 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3785a9-0b0e-47e8-92e0-dd2a92a46372 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:01.677175 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3785a9-0b0e-47e8-92e0-dd2a92a46372 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:01.677448 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3785a9-0b0e-47e8-92e0-dd2a92a46372 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:01.678147 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3785a9-0b0e-47e8-92e0-dd2a92a46372 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:01.679712 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3785a9-0b0e-47e8-92e0-dd2a92a46372 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:01.690551 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1429/2869] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:32:01 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:32:01.694564 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd2f118-48c6-48d8-8941-06b51f6bf8c3 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:01.704855 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd2f118-48c6-48d8-8941-06b51f6bf8c3 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:01.709348 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186fc696-a950-41a7-98c2-22ca72298edb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:01.709959 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186fc696-a950-41a7-98c2-22ca72298edb admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:01.710235 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186fc696-a950-41a7-98c2-22ca72298edb admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:01.710501 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186fc696-a950-41a7-98c2-22ca72298edb admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:01.711132 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186fc696-a950-41a7-98c2-22ca72298edb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:01.712688 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186fc696-a950-41a7-98c2-22ca72298edb admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:01.722329 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1441/2870] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:32:01 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:32:01.726224 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3785a9-0b0e-47e8-92e0-dd2a92a46372 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:01.735860 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3785a9-0b0e-47e8-92e0-dd2a92a46372 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:01.740399 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fc628b-fa4a-4f3b-8295-574f3a530f5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:01.741024 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fc628b-fa4a-4f3b-8295-574f3a530f5d admin admin] REQUEST_METHOD: `PUT` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:01.741301 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fc628b-fa4a-4f3b-8295-574f3a530f5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:01.741572 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fc628b-fa4a-4f3b-8295-574f3a530f5d admin admin] PATH_INFO: `/v3/projects/425a74f118684dc3951c7dd4df7bb3c1/users/34517bd0aaf14c08b596eca75e89cd7c/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:01.756076 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92fc628b-fa4a-4f3b-8295-574f3a530f5d admin admin] RBAC: Authorizing `identity:create_grant(project_id=425a74f118684dc3951c7dd4df7bb3c1, user_id=34517bd0aaf14c08b596eca75e89cd7c, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:01.758696 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92fc628b-fa4a-4f3b-8295-574f3a530f5d admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:01.771617 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1430/2871] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:32:01 2022] PUT /identity/v3/projects/425a74f118684dc3951c7dd4df7bb3c1/users/34517bd0aaf14c08b596eca75e89cd7c/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:32:01.782238 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5900ebb-b7a1-4acb-890f-b6efb47ea699 None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:01.782523 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5900ebb-b7a1-4acb-890f-b6efb47ea699 None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:01.782792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5900ebb-b7a1-4acb-890f-b6efb47ea699 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:01.810437 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-f5900ebb-b7a1-4acb-890f-b6efb47ea699 None None] MFA Rules not processed for user `34517bd0aaf14c08b596eca75e89cd7c`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:32:01.811618 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-f5900ebb-b7a1-4acb-890f-b6efb47ea699 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:01.854848 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:32:01.875177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1442/2872] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:32:01 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:32:01.879775 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fc628b-fa4a-4f3b-8295-574f3a530f5d admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:01.913606 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=94,keystone:INSERT=5,keystone:DELETE=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:32:01.942360 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fc628b-fa4a-4f3b-8295-574f3a530f5d admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:01.946912 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300e01f3-da87-40bc-98ec-f5aadd47030f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:01.947499 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300e01f3-da87-40bc-98ec-f5aadd47030f admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:01.947778 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300e01f3-da87-40bc-98ec-f5aadd47030f admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:01.948050 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300e01f3-da87-40bc-98ec-f5aadd47030f admin admin] PATH_INFO: `/v3/projects` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:01.948680 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300e01f3-da87-40bc-98ec-f5aadd47030f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:01.950589 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300e01f3-da87-40bc-98ec-f5aadd47030f admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:01.975392 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1431/2873] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:32:01 2022] POST /identity/v3/projects => generated 373 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:32:01.979785 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5900ebb-b7a1-4acb-890f-b6efb47ea699 None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:01.990786 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5900ebb-b7a1-4acb-890f-b6efb47ea699 None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:02.045763 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15251d2-2000-4344-9052-817c0303e0df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:02.046855 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15251d2-2000-4344-9052-817c0303e0df admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:02.047145 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15251d2-2000-4344-9052-817c0303e0df admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:02.047424 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15251d2-2000-4344-9052-817c0303e0df admin admin] PATH_INFO: `/v3/users` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:02.048042 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d15251d2-2000-4344-9052-817c0303e0df admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:02.049967 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d15251d2-2000-4344-9052-817c0303e0df admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:02.071471 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1443/2874] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:32:01 2022] POST /identity/v3/users => generated 293 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:32:02.075412 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300e01f3-da87-40bc-98ec-f5aadd47030f admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:02.085446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300e01f3-da87-40bc-98ec-f5aadd47030f admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:02.090055 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69648461-7d2d-4deb-b62b-4c7234521d58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:02.090642 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69648461-7d2d-4deb-b62b-4c7234521d58 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:02.090921 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69648461-7d2d-4deb-b62b-4c7234521d58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:02.091190 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69648461-7d2d-4deb-b62b-4c7234521d58 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:02.091824 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69648461-7d2d-4deb-b62b-4c7234521d58 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:02.093388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69648461-7d2d-4deb-b62b-4c7234521d58 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:02.103557 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1432/2875] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:32:02 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 04:32:02.107625 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15251d2-2000-4344-9052-817c0303e0df admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:02.117727 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15251d2-2000-4344-9052-817c0303e0df admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:02.122308 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f6da1e-04ea-42a5-87f4-0cb467359be7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:02.122947 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f6da1e-04ea-42a5-87f4-0cb467359be7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:02.123225 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f6da1e-04ea-42a5-87f4-0cb467359be7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:02.123497 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f6da1e-04ea-42a5-87f4-0cb467359be7 admin admin] PATH_INFO: `/v3/projects/0e062570862c4d3793aba5c24486c183/users/004a682e7f4740a9bdedbe8ccdccdbc8/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:02.138406 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f6da1e-04ea-42a5-87f4-0cb467359be7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0e062570862c4d3793aba5c24486c183, user_id=004a682e7f4740a9bdedbe8ccdccdbc8, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:02.141036 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f6da1e-04ea-42a5-87f4-0cb467359be7 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:02.153640 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1444/2876] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:32:02 2022] PUT /identity/v3/projects/0e062570862c4d3793aba5c24486c183/users/004a682e7f4740a9bdedbe8ccdccdbc8/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:32:02.157712 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69648461-7d2d-4deb-b62b-4c7234521d58 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:02.167593 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69648461-7d2d-4deb-b62b-4c7234521d58 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:02.225141 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c88d3f-bbae-4e66-a718-5e36756acf48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:02.225758 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c88d3f-bbae-4e66-a718-5e36756acf48 admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:02.226047 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c88d3f-bbae-4e66-a718-5e36756acf48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:02.226320 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c88d3f-bbae-4e66-a718-5e36756acf48 admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:02.226952 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c88d3f-bbae-4e66-a718-5e36756acf48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:02.228522 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c88d3f-bbae-4e66-a718-5e36756acf48 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:02.238489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1433/2877] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:32:02 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:32:02.242602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f6da1e-04ea-42a5-87f4-0cb467359be7 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:02.253170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f6da1e-04ea-42a5-87f4-0cb467359be7 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:02.257695 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f82e55a-7fd0-4e2f-8722-6d449261af33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:02.258360 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f82e55a-7fd0-4e2f-8722-6d449261af33 admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:02.258647 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f82e55a-7fd0-4e2f-8722-6d449261af33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:02.258923 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f82e55a-7fd0-4e2f-8722-6d449261af33 admin admin] PATH_INFO: `/v3/projects/0e062570862c4d3793aba5c24486c183/users/004a682e7f4740a9bdedbe8ccdccdbc8/roles/e6a31562a18649f0809202c36a8e18ef` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:02.260801 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f82e55a-7fd0-4e2f-8722-6d449261af33 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0e062570862c4d3793aba5c24486c183, user_id=004a682e7f4740a9bdedbe8ccdccdbc8, role_id=e6a31562a18649f0809202c36a8e18ef)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:02.263429 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f82e55a-7fd0-4e2f-8722-6d449261af33 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:02.273314 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1445/2878] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:32:02 2022] PUT /identity/v3/projects/0e062570862c4d3793aba5c24486c183/users/004a682e7f4740a9bdedbe8ccdccdbc8/roles/e6a31562a18649f0809202c36a8e18ef => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:32:02.283997 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f8e11e-1b45-4f9b-b95f-401e7370bb0f None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:02.284287 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f8e11e-1b45-4f9b-b95f-401e7370bb0f None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:02.284563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f8e11e-1b45-4f9b-b95f-401e7370bb0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:02.311826 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-b9f8e11e-1b45-4f9b-b95f-401e7370bb0f None None] MFA Rules not processed for user `004a682e7f4740a9bdedbe8ccdccdbc8`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:32:02.313039 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-b9f8e11e-1b45-4f9b-b95f-401e7370bb0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:02.378866 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1434/2879] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:32:02 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:32:02.390125 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f82e55a-7fd0-4e2f-8722-6d449261af33 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:02.391840 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-6f82e55a-7fd0-4e2f-8722-6d449261af33 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:02.447982 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f82e55a-7fd0-4e2f-8722-6d449261af33 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:02.451585 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722c0025-27e5-4d3d-b270-43cd89318e1f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:02.452173 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722c0025-27e5-4d3d-b270-43cd89318e1f service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:02.452453 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722c0025-27e5-4d3d-b270-43cd89318e1f service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:02.452772 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722c0025-27e5-4d3d-b270-43cd89318e1f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:02.503663 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:32:02.508737 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722c0025-27e5-4d3d-b270-43cd89318e1f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:02.510667 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722c0025-27e5-4d3d-b270-43cd89318e1f service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:02.519562 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1446/2880] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:32:02 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:32:04.353822 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f8e11e-1b45-4f9b-b95f-401e7370bb0f None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:04.355232 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-b9f8e11e-1b45-4f9b-b95f-401e7370bb0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:04.367068 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f8e11e-1b45-4f9b-b95f-401e7370bb0f None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:04.370697 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f18e485-783f-467e-90d3-6097538575c0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:04.371290 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f18e485-783f-467e-90d3-6097538575c0 service cinder] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:04.371571 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f18e485-783f-467e-90d3-6097538575c0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:04.371844 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f18e485-783f-467e-90d3-6097538575c0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:04.401174 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:32:04.432978 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f18e485-783f-467e-90d3-6097538575c0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:04.434419 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f18e485-783f-467e-90d3-6097538575c0 service cinder] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:04.443157 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1435/2881] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:32:04 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:32:04.962963 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722c0025-27e5-4d3d-b270-43cd89318e1f service cinder] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:04.972126 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722c0025-27e5-4d3d-b270-43cd89318e1f service cinder] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:04.975711 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc17809-1861-44dc-a6bf-19fc91b6ee18 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:04.976298 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc17809-1861-44dc-a6bf-19fc91b6ee18 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:04.976581 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc17809-1861-44dc-a6bf-19fc91b6ee18 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:04.976854 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc17809-1861-44dc-a6bf-19fc91b6ee18 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:04.985638 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc17809-1861-44dc-a6bf-19fc91b6ee18 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:04.987062 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc17809-1861-44dc-a6bf-19fc91b6ee18 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:04.996200 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1447/2882] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:32:04 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:32:08.864376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f18e485-783f-467e-90d3-6097538575c0 service cinder] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:08.924683 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f18e485-783f-467e-90d3-6097538575c0 service cinder] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:08.929127 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bb3204-d6ca-45fc-a643-c6ffe7dde181 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:08.929737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bb3204-d6ca-45fc-a643-c6ffe7dde181 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:08.930052 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bb3204-d6ca-45fc-a643-c6ffe7dde181 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:08.930323 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bb3204-d6ca-45fc-a643-c6ffe7dde181 admin admin] PATH_INFO: `/v3/users/8859df7b5d7f45c6bb2fad6896c87539` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:08.931418 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bb3204-d6ca-45fc-a643-c6ffe7dde181 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8859df7b5d7f45c6bb2fad6896c87539)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:08.933303 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bb3204-d6ca-45fc-a643-c6ffe7dde181 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:08.976408 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-13bb3204-d6ca-45fc-a643-c6ffe7dde181 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8859df7b5d7f45c6bb2fad6896c87539'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:08.984003 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-13bb3204-d6ca-45fc-a643-c6ffe7dde181 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8859df7b5d7f45c6bb2fad6896c87539'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:08.984681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-13bb3204-d6ca-45fc-a643-c6ffe7dde181 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8859df7b5d7f45c6bb2fad6896c87539'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:08.991811 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-13bb3204-d6ca-45fc-a643-c6ffe7dde181 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8859df7b5d7f45c6bb2fad6896c87539'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:09.004302 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-13bb3204-d6ca-45fc-a643-c6ffe7dde181 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8859df7b5d7f45c6bb2fad6896c87539'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:09.010062 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1436/2883] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:32:08 2022] DELETE /identity/v3/users/8859df7b5d7f45c6bb2fad6896c87539 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:32:09.014589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488e85b1-aa17-4c69-aa7a-c3af9fee5bfd None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:09.015094 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488e85b1-aa17-4c69-aa7a-c3af9fee5bfd None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:09.015196 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488e85b1-aa17-4c69-aa7a-c3af9fee5bfd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:09.034381 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-488e85b1-aa17-4c69-aa7a-c3af9fee5bfd None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:32:09.035558 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-488e85b1-aa17-4c69-aa7a-c3af9fee5bfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:09.092550 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1448/2884] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:32:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:32:09.102294 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bb3204-d6ca-45fc-a643-c6ffe7dde181 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:09.103699 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-13bb3204-d6ca-45fc-a643-c6ffe7dde181 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:09.155370 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:32:09.162227 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bb3204-d6ca-45fc-a643-c6ffe7dde181 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:09.166094 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fff1857-a9b8-4004-9dd7-2d83a6e31cb7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:09.166696 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fff1857-a9b8-4004-9dd7-2d83a6e31cb7 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:09.167051 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fff1857-a9b8-4004-9dd7-2d83a6e31cb7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:09.167352 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fff1857-a9b8-4004-9dd7-2d83a6e31cb7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:09.176763 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fff1857-a9b8-4004-9dd7-2d83a6e31cb7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:09.178198 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fff1857-a9b8-4004-9dd7-2d83a6e31cb7 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:09.187500 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1437/2885] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:32:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:32:09.443424 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488e85b1-aa17-4c69-aa7a-c3af9fee5bfd None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:09.444844 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-488e85b1-aa17-4c69-aa7a-c3af9fee5bfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:09.457343 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488e85b1-aa17-4c69-aa7a-c3af9fee5bfd None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:09.461979 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfea554-b616-4dad-b6f4-4d4df5e0acef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:09.462612 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfea554-b616-4dad-b6f4-4d4df5e0acef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:09.462899 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfea554-b616-4dad-b6f4-4d4df5e0acef admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:09.463159 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfea554-b616-4dad-b6f4-4d4df5e0acef admin admin] PATH_INFO: `/v3/projects/710392938fb8429b991aab0d48faf1d7` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:09.464724 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfea554-b616-4dad-b6f4-4d4df5e0acef admin admin] RBAC: Authorizing `identity:delete_project(project_id=710392938fb8429b991aab0d48faf1d7)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:09.466167 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfea554-b616-4dad-b6f4-4d4df5e0acef admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:09.512302 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-ddfea554-b616-4dad-b6f4-4d4df5e0acef admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '710392938fb8429b991aab0d48faf1d7', 'request_id': 'req-ddfea554-b616-4dad-b6f4-4d4df5e0acef'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:09.518336 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-ddfea554-b616-4dad-b6f4-4d4df5e0acef admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '710392938fb8429b991aab0d48faf1d7', 'request_id': 'req-ddfea554-b616-4dad-b6f4-4d4df5e0acef'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:09.524792 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-ddfea554-b616-4dad-b6f4-4d4df5e0acef admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '710392938fb8429b991aab0d48faf1d7', 'request_id': 'req-ddfea554-b616-4dad-b6f4-4d4df5e0acef'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:09.537929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-ddfea554-b616-4dad-b6f4-4d4df5e0acef admin admin] The token cache is being invalidate because project 710392938fb8429b991aab0d48faf1d7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:32:09.538215 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-ddfea554-b616-4dad-b6f4-4d4df5e0acef admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:09.538851 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-ddfea554-b616-4dad-b6f4-4d4df5e0acef admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:09.540306 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1449/2886] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:32:09 2022] DELETE /identity/v3/projects/710392938fb8429b991aab0d48faf1d7 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:32:09.544379 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fff1857-a9b8-4004-9dd7-2d83a6e31cb7 service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:09.554019 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fff1857-a9b8-4004-9dd7-2d83a6e31cb7 service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:09.614483 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2dfac30-4bee-46c5-86d5-6dd875fcb667 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:09.615116 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2dfac30-4bee-46c5-86d5-6dd875fcb667 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:09.615397 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2dfac30-4bee-46c5-86d5-6dd875fcb667 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:09.615672 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2dfac30-4bee-46c5-86d5-6dd875fcb667 admin admin] PATH_INFO: `/v3/users/34517bd0aaf14c08b596eca75e89cd7c` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:09.616776 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2dfac30-4bee-46c5-86d5-6dd875fcb667 admin admin] RBAC: Authorizing `identity:delete_user(user_id=34517bd0aaf14c08b596eca75e89cd7c)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:09.618710 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2dfac30-4bee-46c5-86d5-6dd875fcb667 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:09.664288 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-f2dfac30-4bee-46c5-86d5-6dd875fcb667 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '34517bd0aaf14c08b596eca75e89cd7c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:09.670948 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-f2dfac30-4bee-46c5-86d5-6dd875fcb667 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '34517bd0aaf14c08b596eca75e89cd7c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:09.671609 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-f2dfac30-4bee-46c5-86d5-6dd875fcb667 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '34517bd0aaf14c08b596eca75e89cd7c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:09.678953 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-f2dfac30-4bee-46c5-86d5-6dd875fcb667 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '34517bd0aaf14c08b596eca75e89cd7c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:09.690723 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-f2dfac30-4bee-46c5-86d5-6dd875fcb667 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '34517bd0aaf14c08b596eca75e89cd7c'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:09.696371 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1438/2887] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:32:09 2022] DELETE /identity/v3/users/34517bd0aaf14c08b596eca75e89cd7c => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:32:09.972049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfea554-b616-4dad-b6f4-4d4df5e0acef admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:09.973509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-ddfea554-b616-4dad-b6f4-4d4df5e0acef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:09.987512 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfea554-b616-4dad-b6f4-4d4df5e0acef admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:10.043548 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948c32e8-2b6a-44c8-bf5e-da44132f2353 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:10.044214 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948c32e8-2b6a-44c8-bf5e-da44132f2353 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:10.044499 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948c32e8-2b6a-44c8-bf5e-da44132f2353 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:10.044798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948c32e8-2b6a-44c8-bf5e-da44132f2353 admin admin] PATH_INFO: `/v3/projects/425a74f118684dc3951c7dd4df7bb3c1` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:10.045969 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948c32e8-2b6a-44c8-bf5e-da44132f2353 admin admin] RBAC: Authorizing `identity:delete_project(project_id=425a74f118684dc3951c7dd4df7bb3c1)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:10.047862 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948c32e8-2b6a-44c8-bf5e-da44132f2353 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:10.053929 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:32:10.095182 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-948c32e8-2b6a-44c8-bf5e-da44132f2353 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '425a74f118684dc3951c7dd4df7bb3c1', 'request_id': 'req-948c32e8-2b6a-44c8-bf5e-da44132f2353'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:10.100692 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-948c32e8-2b6a-44c8-bf5e-da44132f2353 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '425a74f118684dc3951c7dd4df7bb3c1', 'request_id': 'req-948c32e8-2b6a-44c8-bf5e-da44132f2353'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:10.106753 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-948c32e8-2b6a-44c8-bf5e-da44132f2353 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '425a74f118684dc3951c7dd4df7bb3c1', 'request_id': 'req-948c32e8-2b6a-44c8-bf5e-da44132f2353'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:10.118858 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-948c32e8-2b6a-44c8-bf5e-da44132f2353 admin admin] The token cache is being invalidate because project 425a74f118684dc3951c7dd4df7bb3c1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:32:10.119142 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-948c32e8-2b6a-44c8-bf5e-da44132f2353 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:10.119774 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-948c32e8-2b6a-44c8-bf5e-da44132f2353 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:10.121272 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1450/2888] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:32:09 2022] DELETE /identity/v3/projects/425a74f118684dc3951c7dd4df7bb3c1 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:32:10.127792 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2dfac30-4bee-46c5-86d5-6dd875fcb667 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:10.129248 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-f2dfac30-4bee-46c5-86d5-6dd875fcb667 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:10.144616 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2dfac30-4bee-46c5-86d5-6dd875fcb667 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:10.200160 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a949bf65-f0f9-4184-805e-fbeff3ba5469 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:10.200691 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a949bf65-f0f9-4184-805e-fbeff3ba5469 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:10.200972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a949bf65-f0f9-4184-805e-fbeff3ba5469 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:10.201247 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a949bf65-f0f9-4184-805e-fbeff3ba5469 admin admin] PATH_INFO: `/v3/users/004a682e7f4740a9bdedbe8ccdccdbc8` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:10.202376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a949bf65-f0f9-4184-805e-fbeff3ba5469 admin admin] RBAC: Authorizing `identity:delete_user(user_id=004a682e7f4740a9bdedbe8ccdccdbc8)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:10.204388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a949bf65-f0f9-4184-805e-fbeff3ba5469 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:10.215841 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:32:10.252407 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a949bf65-f0f9-4184-805e-fbeff3ba5469 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '004a682e7f4740a9bdedbe8ccdccdbc8'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:10.259153 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a949bf65-f0f9-4184-805e-fbeff3ba5469 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '004a682e7f4740a9bdedbe8ccdccdbc8'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:10.259813 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a949bf65-f0f9-4184-805e-fbeff3ba5469 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '004a682e7f4740a9bdedbe8ccdccdbc8'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:10.266941 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a949bf65-f0f9-4184-805e-fbeff3ba5469 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '004a682e7f4740a9bdedbe8ccdccdbc8'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:10.278963 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-a949bf65-f0f9-4184-805e-fbeff3ba5469 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '004a682e7f4740a9bdedbe8ccdccdbc8'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:10.285114 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1439/2889] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:32:10 2022] DELETE /identity/v3/users/004a682e7f4740a9bdedbe8ccdccdbc8 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:32:10.590224 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948c32e8-2b6a-44c8-bf5e-da44132f2353 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:10.591633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-948c32e8-2b6a-44c8-bf5e-da44132f2353 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:10.604276 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948c32e8-2b6a-44c8-bf5e-da44132f2353 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:10.668269 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe06ef6a-b61c-4781-8e32-a49c3e97d2ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:10.669456 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe06ef6a-b61c-4781-8e32-a49c3e97d2ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:10.669790 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe06ef6a-b61c-4781-8e32-a49c3e97d2ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:10.670633 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe06ef6a-b61c-4781-8e32-a49c3e97d2ca admin admin] PATH_INFO: `/v3/projects/0e062570862c4d3793aba5c24486c183` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:10.671850 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe06ef6a-b61c-4781-8e32-a49c3e97d2ca admin admin] RBAC: Authorizing `identity:delete_project(project_id=0e062570862c4d3793aba5c24486c183)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:10.674261 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe06ef6a-b61c-4781-8e32-a49c3e97d2ca admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:10.721620 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-fe06ef6a-b61c-4781-8e32-a49c3e97d2ca admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0e062570862c4d3793aba5c24486c183', 'request_id': 'req-fe06ef6a-b61c-4781-8e32-a49c3e97d2ca'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:10.727247 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-fe06ef6a-b61c-4781-8e32-a49c3e97d2ca admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0e062570862c4d3793aba5c24486c183', 'request_id': 'req-fe06ef6a-b61c-4781-8e32-a49c3e97d2ca'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:10.733824 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-fe06ef6a-b61c-4781-8e32-a49c3e97d2ca admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0e062570862c4d3793aba5c24486c183', 'request_id': 'req-fe06ef6a-b61c-4781-8e32-a49c3e97d2ca'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:10.745888 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-fe06ef6a-b61c-4781-8e32-a49c3e97d2ca admin admin] The token cache is being invalidate because project 0e062570862c4d3793aba5c24486c183 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81420) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:32:10.746170 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-fe06ef6a-b61c-4781-8e32-a49c3e97d2ca admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:10.746783 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-fe06ef6a-b61c-4781-8e32-a49c3e97d2ca admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:32:10.748257 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1451/2890] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:32:10 2022] DELETE /identity/v3/projects/0e062570862c4d3793aba5c24486c183 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:32:10.753098 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d5b699-afc2-4ae3-ab60-6809bff3e02d None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:10.753509 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d5b699-afc2-4ae3-ab60-6809bff3e02d None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:10.753813 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d5b699-afc2-4ae3-ab60-6809bff3e02d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:10.772727 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-f4d5b699-afc2-4ae3-ab60-6809bff3e02d None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:32:10.774210 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-f4d5b699-afc2-4ae3-ab60-6809bff3e02d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:10.830670 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1440/2891] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:32:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:32:10.838521 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b4a804-0475-4f81-b899-54ae2c789f8f None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:10.838802 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b4a804-0475-4f81-b899-54ae2c789f8f None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:10.839072 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b4a804-0475-4f81-b899-54ae2c789f8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:10.859234 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-d3b4a804-0475-4f81-b899-54ae2c789f8f None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:32:10.860436 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-d3b4a804-0475-4f81-b899-54ae2c789f8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:10.868048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1452/2892] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:32:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:32:10.872006 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d5b699-afc2-4ae3-ab60-6809bff3e02d None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:10.881351 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d5b699-afc2-4ae3-ab60-6809bff3e02d None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:10.885812 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bed379-7b2b-43a2-8d77-2dcaa123ce2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:10.886396 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bed379-7b2b-43a2-8d77-2dcaa123ce2b admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:10.886669 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bed379-7b2b-43a2-8d77-2dcaa123ce2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:10.886940 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bed379-7b2b-43a2-8d77-2dcaa123ce2b admin admin] PATH_INFO: `/v3/domains` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:10.887577 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9bed379-7b2b-43a2-8d77-2dcaa123ce2b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81421) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 04:32:10.887875 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9bed379-7b2b-43a2-8d77-2dcaa123ce2b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:10.889457 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9bed379-7b2b-43a2-8d77-2dcaa123ce2b admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:10.900851 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1441/2893] 10.222.0.26 () {58 vars in 1153 bytes} [Tue Sep 27 04:32:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 04:32:10.905639 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b4a804-0475-4f81-b899-54ae2c789f8f None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:10.914683 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b4a804-0475-4f81-b899-54ae2c789f8f None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:10.919245 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ef439f-3611-4e91-bdf5-20c3a109ecf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:10.920423 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ef439f-3611-4e91-bdf5-20c3a109ecf8 admin admin] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:10.920706 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ef439f-3611-4e91-bdf5-20c3a109ecf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:10.920974 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ef439f-3611-4e91-bdf5-20c3a109ecf8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:10.921589 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ef439f-3611-4e91-bdf5-20c3a109ecf8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:10.923484 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ef439f-3611-4e91-bdf5-20c3a109ecf8 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:10.938543 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:32:10.950823 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1453/2894] 10.222.0.26 () {60 vars in 1153 bytes} [Tue Sep 27 04:32:10 2022] POST /identity/v3/projects => generated 371 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 04:32:10.955411 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bed379-7b2b-43a2-8d77-2dcaa123ce2b admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:10.965276 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bed379-7b2b-43a2-8d77-2dcaa123ce2b admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:11.021039 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731c04c0-4bd4-4c70-a283-7d943a7e97d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:11.021583 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731c04c0-4bd4-4c70-a283-7d943a7e97d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:11.021871 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731c04c0-4bd4-4c70-a283-7d943a7e97d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:11.022152 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731c04c0-4bd4-4c70-a283-7d943a7e97d4 admin admin] PATH_INFO: `/v3/users` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:11.022783 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731c04c0-4bd4-4c70-a283-7d943a7e97d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:11.024938 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731c04c0-4bd4-4c70-a283-7d943a7e97d4 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:11.046073 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1442/2895] 10.222.0.26 () {60 vars in 1144 bytes} [Tue Sep 27 04:32:10 2022] POST /identity/v3/users => generated 286 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 04:32:11.050323 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ef439f-3611-4e91-bdf5-20c3a109ecf8 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:11.061136 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ef439f-3611-4e91-bdf5-20c3a109ecf8 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:11.065696 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275866e6-6f3e-4afe-b5b1-db75cdeef42d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:11.066325 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275866e6-6f3e-4afe-b5b1-db75cdeef42d admin admin] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:11.066580 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275866e6-6f3e-4afe-b5b1-db75cdeef42d admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:11.066847 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275866e6-6f3e-4afe-b5b1-db75cdeef42d admin admin] PATH_INFO: `/v3/roles` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:11.067553 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275866e6-6f3e-4afe-b5b1-db75cdeef42d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:11.069115 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275866e6-6f3e-4afe-b5b1-db75cdeef42d admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:11.078816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1454/2896] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:32:11 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:32:11.082926 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731c04c0-4bd4-4c70-a283-7d943a7e97d4 admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:11.089198 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:DELETE=10,keystone:SELECT=86,keystone:INSERT=4 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:32:11.093105 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731c04c0-4bd4-4c70-a283-7d943a7e97d4 admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:11.098681 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edd1b32-4dd9-46b7-8888-3afb24120cff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:11.099434 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edd1b32-4dd9-46b7-8888-3afb24120cff admin admin] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:11.099889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edd1b32-4dd9-46b7-8888-3afb24120cff admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:11.100327 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edd1b32-4dd9-46b7-8888-3afb24120cff admin admin] PATH_INFO: `/v3/roles` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:11.100968 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edd1b32-4dd9-46b7-8888-3afb24120cff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:11.102560 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edd1b32-4dd9-46b7-8888-3afb24120cff admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:11.113388 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1443/2897] 10.222.0.26 () {58 vars in 1122 bytes} [Tue Sep 27 04:32:11 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 04:32:11.117333 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275866e6-6f3e-4afe-b5b1-db75cdeef42d admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:11.127498 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275866e6-6f3e-4afe-b5b1-db75cdeef42d admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:11.132034 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7655ce6-90d7-401a-91b6-4ea6dac9abfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:11.132664 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7655ce6-90d7-401a-91b6-4ea6dac9abfa admin admin] REQUEST_METHOD: `PUT` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:11.132937 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7655ce6-90d7-401a-91b6-4ea6dac9abfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:11.133205 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7655ce6-90d7-401a-91b6-4ea6dac9abfa admin admin] PATH_INFO: `/v3/projects/494bdfe04cfa4dfab2c7ab389a889298/users/20bf50d405e84ca7abd123e3c7a98c69/roles/e3ce5a3697ec4292a446366d3caadeea` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:11.147281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7655ce6-90d7-401a-91b6-4ea6dac9abfa admin admin] RBAC: Authorizing `identity:create_grant(project_id=494bdfe04cfa4dfab2c7ab389a889298, user_id=20bf50d405e84ca7abd123e3c7a98c69, role_id=e3ce5a3697ec4292a446366d3caadeea)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:11.150177 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7655ce6-90d7-401a-91b6-4ea6dac9abfa admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:11.162990 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1455/2898] 10.222.0.26 () {60 vars in 1483 bytes} [Tue Sep 27 04:32:11 2022] PUT /identity/v3/projects/494bdfe04cfa4dfab2c7ab389a889298/users/20bf50d405e84ca7abd123e3c7a98c69/roles/e3ce5a3697ec4292a446366d3caadeea => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 04:32:11.168132 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8467119-ab1d-45ca-ad22-0d1f97c4dbb2 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:11.168612 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8467119-ab1d-45ca-ad22-0d1f97c4dbb2 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:11.168737 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8467119-ab1d-45ca-ad22-0d1f97c4dbb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:11.188449 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-c8467119-ab1d-45ca-ad22-0d1f97c4dbb2 None None] MFA Rules not processed for user `6190ca8b8acd4e99b228f0851e2ed5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:32:11.189690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-c8467119-ab1d-45ca-ad22-0d1f97c4dbb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:11.247007 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1444/2899] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:32:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:32:11.257132 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7655ce6-90d7-401a-91b6-4ea6dac9abfa admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:11.258806 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-f7655ce6-90d7-401a-91b6-4ea6dac9abfa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:11.314247 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7655ce6-90d7-401a-91b6-4ea6dac9abfa admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:11.317716 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f97c0b-c2a5-430d-beed-662bfeed8115 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:11.318330 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f97c0b-c2a5-430d-beed-662bfeed8115 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:11.318602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f97c0b-c2a5-430d-beed-662bfeed8115 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:11.318871 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f97c0b-c2a5-430d-beed-662bfeed8115 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:11.327531 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f97c0b-c2a5-430d-beed-662bfeed8115 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:11.328925 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f97c0b-c2a5-430d-beed-662bfeed8115 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:11.338049 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1456/2900] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:32:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:32:11.956368 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9361b13e-b0a0-4a29-8f2f-72eb2066d959 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:11.956690 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9361b13e-b0a0-4a29-8f2f-72eb2066d959 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:11.956968 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9361b13e-b0a0-4a29-8f2f-72eb2066d959 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:11.986195 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-9361b13e-b0a0-4a29-8f2f-72eb2066d959 None None] MFA Rules not processed for user `20bf50d405e84ca7abd123e3c7a98c69`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:32:11.987438 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-9361b13e-b0a0-4a29-8f2f-72eb2066d959 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:12.054707 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1445/2901] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:32:11 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:32:12.065843 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f97c0b-c2a5-430d-beed-662bfeed8115 service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:12.067199 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-87f97c0b-c2a5-430d-beed-662bfeed8115 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:12.125816 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f97c0b-c2a5-430d-beed-662bfeed8115 service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:12.129560 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6482a021-1201-4e7c-8ab3-630014682157 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3970310fc7e34704b686e8d3bff3fdb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:12.130190 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6482a021-1201-4e7c-8ab3-630014682157 service cinder] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:12.130470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6482a021-1201-4e7c-8ab3-630014682157 service cinder] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:12.130743 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6482a021-1201-4e7c-8ab3-630014682157 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:12.139740 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6482a021-1201-4e7c-8ab3-630014682157 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:12.141140 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6482a021-1201-4e7c-8ab3-630014682157 service cinder] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:12.150030 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1457/2902] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:32:12 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:32:13.517868 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec166f4e-ee6b-4753-b0a3-02f944e9e3b2 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:13.518402 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec166f4e-ee6b-4753-b0a3-02f944e9e3b2 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:13.518402 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec166f4e-ee6b-4753-b0a3-02f944e9e3b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:13.540492 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-ec166f4e-ee6b-4753-b0a3-02f944e9e3b2 None None] MFA Rules not processed for user `20bf50d405e84ca7abd123e3c7a98c69`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:32:13.541842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ec166f4e-ee6b-4753-b0a3-02f944e9e3b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:13.549598 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1446/2903] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:32:13 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:32:14.095034 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084daf4e-4d65-4621-bcc3-606e025102da None None] REQUEST_METHOD: `POST` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:14.095432 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084daf4e-4d65-4621-bcc3-606e025102da None None] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:14.095624 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084daf4e-4d65-4621-bcc3-606e025102da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:14.120260 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.auth.core [None req-084daf4e-4d65-4621-bcc3-606e025102da None None] MFA Rules not processed for user `44a710d801334b20b9f433eaad61cbc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81420) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:32:14.121490 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-084daf4e-4d65-4621-bcc3-606e025102da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:14.170445 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec166f4e-ee6b-4753-b0a3-02f944e9e3b2 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:14.172370 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-ec166f4e-ee6b-4753-b0a3-02f944e9e3b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:14.198081 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1458/2904] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:32:14 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 04:32:14.252207 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec166f4e-ee6b-4753-b0a3-02f944e9e3b2 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:14.257344 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c27b29c-1c89-4574-ab2d-9dd46fc065d2 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3a1bc198dfc0486e9e56d4fb28cda05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:14.258012 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c27b29c-1c89-4574-ab2d-9dd46fc065d2 service placement] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:14.258293 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c27b29c-1c89-4574-ab2d-9dd46fc065d2 service placement] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:14.258564 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c27b29c-1c89-4574-ab2d-9dd46fc065d2 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:14.260277 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-3c27b29c-1c89-4574-ab2d-9dd46fc065d2 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:14.272376 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c27b29c-1c89-4574-ab2d-9dd46fc065d2 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:14.274343 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c27b29c-1c89-4574-ab2d-9dd46fc065d2 service placement] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:14.284965 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1447/2905] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:32:14 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:32:23.321457 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084daf4e-4d65-4621-bcc3-606e025102da None None] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:23.322894 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-084daf4e-4d65-4621-bcc3-606e025102da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:23.337356 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:32:23.398445 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084daf4e-4d65-4621-bcc3-606e025102da None None] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:23.404393 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55abde7f-2b3a-4f59-9686-6cc6c8193f48 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:23.405001 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55abde7f-2b3a-4f59-9686-6cc6c8193f48 service nova] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:23.405281 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55abde7f-2b3a-4f59-9686-6cc6c8193f48 service nova] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:23.405556 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55abde7f-2b3a-4f59-9686-6cc6c8193f48 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:23.407650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-55abde7f-2b3a-4f59-9686-6cc6c8193f48 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:23.421094 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55abde7f-2b3a-4f59-9686-6cc6c8193f48 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:23.422725 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55abde7f-2b3a-4f59-9686-6cc6c8193f48 service nova] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:23.433217 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1459/2906] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:32:23 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:32:24.280621 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=97 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:32:30.391214 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e52dcf1-aeb4-4e03-9e34-66c31ef120d2 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:30.391807 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e52dcf1-aeb4-4e03-9e34-66c31ef120d2 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:30.391807 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e52dcf1-aeb4-4e03-9e34-66c31ef120d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:30.411473 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-1e52dcf1-aeb4-4e03-9e34-66c31ef120d2 None None] MFA Rules not processed for user `20bf50d405e84ca7abd123e3c7a98c69`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:32:30.412684 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-1e52dcf1-aeb4-4e03-9e34-66c31ef120d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:30.419834 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1448/2907] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:32:30 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:32:33.428746 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=29 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:32:33.535614 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55abde7f-2b3a-4f59-9686-6cc6c8193f48 service nova] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:33.537152 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-55abde7f-2b3a-4f59-9686-6cc6c8193f48 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:33.552617 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55abde7f-2b3a-4f59-9686-6cc6c8193f48 service nova] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:33.560141 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659c0eb3-0d6a-4973-b32b-9433241256b5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a1586ee841cd44658e4d5feb25de7940', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:33.560769 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659c0eb3-0d6a-4973-b32b-9433241256b5 service nova] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:33.561045 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659c0eb3-0d6a-4973-b32b-9433241256b5 service nova] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:33.561313 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659c0eb3-0d6a-4973-b32b-9433241256b5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:33.563178 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-659c0eb3-0d6a-4973-b32b-9433241256b5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:33.578379 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659c0eb3-0d6a-4973-b32b-9433241256b5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:33.580093 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659c0eb3-0d6a-4973-b32b-9433241256b5 service nova] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:33.592704 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1460/2908] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:32:33 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:32:40.405728 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=3 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:32:43.588402 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=6 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:32:47.877553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173dbc6f-f47b-49b1-8269-546b341f219c None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:47.878072 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173dbc6f-f47b-49b1-8269-546b341f219c None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:47.878184 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173dbc6f-f47b-49b1-8269-546b341f219c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:47.938389 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-173dbc6f-f47b-49b1-8269-546b341f219c None None] MFA Rules not processed for user `5a2fcc8c9ac44dceadcc30d163bbfa0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:32:47.939942 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-173dbc6f-f47b-49b1-8269-546b341f219c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:48.002111 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1449/2909] 10.222.0.26 () {58 vars in 1009 bytes} [Tue Sep 27 04:32:47 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:32:48.051154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659c0eb3-0d6a-4973-b32b-9433241256b5 service nova] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:32:48.052665 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-659c0eb3-0d6a-4973-b32b-9433241256b5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:32:48.145331 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659c0eb3-0d6a-4973-b32b-9433241256b5 service nova] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:32:48.150946 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a43998f-6239-4c77-a38a-7ba9bdb6721b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '181dd5dc720c4a97860852062b7b19c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:32:48.151641 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a43998f-6239-4c77-a38a-7ba9bdb6721b service swift] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:32:48.151921 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a43998f-6239-4c77-a38a-7ba9bdb6721b service swift] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:32:48.152194 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a43998f-6239-4c77-a38a-7ba9bdb6721b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:32:48.160457 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a43998f-6239-4c77-a38a-7ba9bdb6721b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:32:48.167623 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a43998f-6239-4c77-a38a-7ba9bdb6721b service swift] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:32:48.176619 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1461/2910] 10.222.0.26 () {60 vars in 1384 bytes} [Tue Sep 27 04:32:48 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 04:32:57.999369 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=44 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:32:58.173407 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=41 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:33:02.571689 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab00848-96fa-459b-8302-2ef6c6881587 None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:33:02.572183 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab00848-96fa-459b-8302-2ef6c6881587 None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:33:02.572310 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab00848-96fa-459b-8302-2ef6c6881587 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:33:02.591528 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-fab00848-96fa-459b-8302-2ef6c6881587 None None] MFA Rules not processed for user `5a2fcc8c9ac44dceadcc30d163bbfa0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:33:02.592722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-fab00848-96fa-459b-8302-2ef6c6881587 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:33:02.599796 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1450/2911] 10.222.0.26 () {58 vars in 967 bytes} [Tue Sep 27 04:33:02 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:33:02.611877 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a43998f-6239-4c77-a38a-7ba9bdb6721b service swift] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:33:02.623798 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a43998f-6239-4c77-a38a-7ba9bdb6721b service swift] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:33:02.628742 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b21bf2-82e9-4d60-9eb4-d7bb9a19a808 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '181dd5dc720c4a97860852062b7b19c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:33:02.629328 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b21bf2-82e9-4d60-9eb4-d7bb9a19a808 service swift] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:33:02.629601 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b21bf2-82e9-4d60-9eb4-d7bb9a19a808 service swift] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:33:02.630525 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b21bf2-82e9-4d60-9eb4-d7bb9a19a808 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:33:02.638272 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b21bf2-82e9-4d60-9eb4-d7bb9a19a808 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:33:02.639810 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b21bf2-82e9-4d60-9eb4-d7bb9a19a808 service swift] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:33:02.648847 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1462/2912] 10.222.0.26 () {60 vars in 1384 bytes} [Tue Sep 27 04:33:02 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 04:33:04.477313 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab00848-96fa-459b-8302-2ef6c6881587 None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:33:04.489814 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab00848-96fa-459b-8302-2ef6c6881587 None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:33:04.494722 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c7c1e5-a091-49d9-bb6d-55d7ba3c5a50 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3a1bc198dfc0486e9e56d4fb28cda05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:33:04.495326 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c7c1e5-a091-49d9-bb6d-55d7ba3c5a50 service placement] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:33:04.495610 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c7c1e5-a091-49d9-bb6d-55d7ba3c5a50 service placement] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:33:04.495883 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c7c1e5-a091-49d9-bb6d-55d7ba3c5a50 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:33:04.497603 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-91c7c1e5-a091-49d9-bb6d-55d7ba3c5a50 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:33:04.509905 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c7c1e5-a091-49d9-bb6d-55d7ba3c5a50 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:33:04.511472 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c7c1e5-a091-49d9-bb6d-55d7ba3c5a50 service placement] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:33:04.522154 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1451/2913] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:33:04 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:33:04.536452 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b21bf2-82e9-4d60-9eb4-d7bb9a19a808 service swift] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:33:04.550593 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b21bf2-82e9-4d60-9eb4-d7bb9a19a808 service swift] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:33:04.556003 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f600b4-e7ae-4312-a91e-441105577a72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:33:04.556728 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f600b4-e7ae-4312-a91e-441105577a72 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:33:04.557074 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f600b4-e7ae-4312-a91e-441105577a72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:33:04.557415 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f600b4-e7ae-4312-a91e-441105577a72 admin admin] PATH_INFO: `/v3/users/20bf50d405e84ca7abd123e3c7a98c69` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:33:04.558648 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f600b4-e7ae-4312-a91e-441105577a72 admin admin] RBAC: Authorizing `identity:delete_user(user_id=20bf50d405e84ca7abd123e3c7a98c69)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:33:04.560531 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f600b4-e7ae-4312-a91e-441105577a72 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:33:04.608254 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-08f600b4-e7ae-4312-a91e-441105577a72 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '20bf50d405e84ca7abd123e3c7a98c69'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:33:04.615693 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-08f600b4-e7ae-4312-a91e-441105577a72 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '20bf50d405e84ca7abd123e3c7a98c69'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:33:04.616341 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-08f600b4-e7ae-4312-a91e-441105577a72 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '20bf50d405e84ca7abd123e3c7a98c69'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:33:04.623455 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-08f600b4-e7ae-4312-a91e-441105577a72 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '20bf50d405e84ca7abd123e3c7a98c69'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:33:04.635385 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-08f600b4-e7ae-4312-a91e-441105577a72 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '20bf50d405e84ca7abd123e3c7a98c69'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:33:04.641535 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1463/2914] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:33:04 2022] DELETE /identity/v3/users/20bf50d405e84ca7abd123e3c7a98c69 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:33:04.807032 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c7c1e5-a091-49d9-bb6d-55d7ba3c5a50 service placement] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:33:04.818647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c7c1e5-a091-49d9-bb6d-55d7ba3c5a50 service placement] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:33:04.874856 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f661c2f-c3d6-4199-a783-804d65f1bb26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:33:04.875471 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f661c2f-c3d6-4199-a783-804d65f1bb26 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:33:04.875825 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f661c2f-c3d6-4199-a783-804d65f1bb26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:33:04.876095 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f661c2f-c3d6-4199-a783-804d65f1bb26 admin admin] PATH_INFO: `/v3/projects/494bdfe04cfa4dfab2c7ab389a889298` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:33:04.877173 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f661c2f-c3d6-4199-a783-804d65f1bb26 admin admin] RBAC: Authorizing `identity:delete_project(project_id=494bdfe04cfa4dfab2c7ab389a889298)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:33:04.879100 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f661c2f-c3d6-4199-a783-804d65f1bb26 admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:33:04.923563 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-5f661c2f-c3d6-4199-a783-804d65f1bb26 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '494bdfe04cfa4dfab2c7ab389a889298', 'request_id': 'req-5f661c2f-c3d6-4199-a783-804d65f1bb26'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:33:04.929529 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-5f661c2f-c3d6-4199-a783-804d65f1bb26 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '494bdfe04cfa4dfab2c7ab389a889298', 'request_id': 'req-5f661c2f-c3d6-4199-a783-804d65f1bb26'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:33:04.935406 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-5f661c2f-c3d6-4199-a783-804d65f1bb26 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '494bdfe04cfa4dfab2c7ab389a889298', 'request_id': 'req-5f661c2f-c3d6-4199-a783-804d65f1bb26'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:33:04.947486 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-5f661c2f-c3d6-4199-a783-804d65f1bb26 admin admin] The token cache is being invalidate because project 494bdfe04cfa4dfab2c7ab389a889298 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:33:04.947777 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-5f661c2f-c3d6-4199-a783-804d65f1bb26 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:33:04.948401 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-5f661c2f-c3d6-4199-a783-804d65f1bb26 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:33:04.949880 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1452/2915] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:33:04 2022] DELETE /identity/v3/projects/494bdfe04cfa4dfab2c7ab389a889298 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:33:12.869619 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f600b4-e7ae-4312-a91e-441105577a72 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:33:12.871048 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-08f600b4-e7ae-4312-a91e-441105577a72 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:33:12.939070 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f600b4-e7ae-4312-a91e-441105577a72 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:33:12.989962 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:33:12.997449 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a071b60d-3d1a-430f-a2a6-7a76e7535ae2 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3a1bc198dfc0486e9e56d4fb28cda05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:33:12.998519 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a071b60d-3d1a-430f-a2a6-7a76e7535ae2 service placement] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:33:12.998892 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a071b60d-3d1a-430f-a2a6-7a76e7535ae2 service placement] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:33:12.999202 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a071b60d-3d1a-430f-a2a6-7a76e7535ae2 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:33:13.001268 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a071b60d-3d1a-430f-a2a6-7a76e7535ae2 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:33:13.012801 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a071b60d-3d1a-430f-a2a6-7a76e7535ae2 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:33:13.014408 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a071b60d-3d1a-430f-a2a6-7a76e7535ae2 service placement] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:33:13.024768 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1464/2916] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:33:12 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:33:14.944027 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=60,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:33:23.020094 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=4 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:34:08.755446 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205445d1-081c-405e-af90-a5ac7bd7681a None None] REQUEST_METHOD: `POST` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:34:08.756030 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205445d1-081c-405e-af90-a5ac7bd7681a None None] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:34:08.756030 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205445d1-081c-405e-af90-a5ac7bd7681a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:34:08.775588 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.auth.core [None req-205445d1-081c-405e-af90-a5ac7bd7681a None None] MFA Rules not processed for user `44a710d801334b20b9f433eaad61cbc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81421) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 04:34:08.776765 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-205445d1-081c-405e-af90-a5ac7bd7681a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:34:08.828909 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1453/2917] 10.222.0.26 () {58 vars in 958 bytes} [Tue Sep 27 04:34:08 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 04:34:09.164982 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a071b60d-3d1a-430f-a2a6-7a76e7535ae2 service placement] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:34:09.166373 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-a071b60d-3d1a-430f-a2a6-7a76e7535ae2 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:34:09.222009 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a071b60d-3d1a-430f-a2a6-7a76e7535ae2 service placement] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:34:09.225635 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873760e1-f2ba-44ea-bb5b-adf32762e964 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:34:09.226278 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873760e1-f2ba-44ea-bb5b-adf32762e964 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:34:09.226554 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873760e1-f2ba-44ea-bb5b-adf32762e964 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:34:09.226824 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873760e1-f2ba-44ea-bb5b-adf32762e964 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:34:09.228529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-873760e1-f2ba-44ea-bb5b-adf32762e964 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:34:09.238597 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873760e1-f2ba-44ea-bb5b-adf32762e964 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:34:09.240004 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873760e1-f2ba-44ea-bb5b-adf32762e964 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:34:09.247843 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1465/2918] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:34:09 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:34:18.822181 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=25 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:34:19.244300 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=26 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:35:56.303187 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205445d1-081c-405e-af90-a5ac7bd7681a None None] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:35:56.313450 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205445d1-081c-405e-af90-a5ac7bd7681a None None] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:35:56.316979 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ec324d-cd3d-4971-a859-bc9c1812a25c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:35:56.317573 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ec324d-cd3d-4971-a859-bc9c1812a25c service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:35:56.317889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ec324d-cd3d-4971-a859-bc9c1812a25c service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:35:56.318165 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ec324d-cd3d-4971-a859-bc9c1812a25c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:35:56.319863 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-47ec324d-cd3d-4971-a859-bc9c1812a25c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:35:56.330448 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ec324d-cd3d-4971-a859-bc9c1812a25c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:35:56.331842 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ec324d-cd3d-4971-a859-bc9c1812a25c service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:35:56.340526 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1454/2919] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:35:56 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 04:35:57.543682 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873760e1-f2ba-44ea-bb5b-adf32762e964 service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:35:57.545105 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-873760e1-f2ba-44ea-bb5b-adf32762e964 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:35:57.556754 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873760e1-f2ba-44ea-bb5b-adf32762e964 service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:35:57.561090 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28506988-9352-4c43-985a-b33f6055e451 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:35:57.561711 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28506988-9352-4c43-985a-b33f6055e451 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:35:57.562172 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28506988-9352-4c43-985a-b33f6055e451 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:35:57.562450 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28506988-9352-4c43-985a-b33f6055e451 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:35:57.564441 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-28506988-9352-4c43-985a-b33f6055e451 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:35:57.574856 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28506988-9352-4c43-985a-b33f6055e451 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:35:57.576420 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28506988-9352-4c43-985a-b33f6055e451 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:35:57.585602 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1466/2920] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:35:57 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:35:57.661639 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ec324d-cd3d-4971-a859-bc9c1812a25c service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:35:57.671205 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ec324d-cd3d-4971-a859-bc9c1812a25c service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:35:57.674907 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a1cba7-eea6-431c-84bc-d84a8c89c784 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:35:57.675531 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a1cba7-eea6-431c-84bc-d84a8c89c784 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:35:57.675809 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a1cba7-eea6-431c-84bc-d84a8c89c784 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:35:57.676084 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a1cba7-eea6-431c-84bc-d84a8c89c784 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:35:57.677803 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-a3a1cba7-eea6-431c-84bc-d84a8c89c784 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:35:57.753431 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a1cba7-eea6-431c-84bc-d84a8c89c784 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:35:57.754889 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a1cba7-eea6-431c-84bc-d84a8c89c784 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:35:57.766256 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1455/2921] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:35:57 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:35:58.957380 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28506988-9352-4c43-985a-b33f6055e451 service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:35:58.965945 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28506988-9352-4c43-985a-b33f6055e451 service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:35:58.969474 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc4891c-ecef-404d-91a8-d730d1ecddb8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:35:58.970109 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc4891c-ecef-404d-91a8-d730d1ecddb8 service neutron] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:35:58.970387 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc4891c-ecef-404d-91a8-d730d1ecddb8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:35:58.970659 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc4891c-ecef-404d-91a8-d730d1ecddb8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:35:58.972352 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-7bc4891c-ecef-404d-91a8-d730d1ecddb8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:35:58.985280 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc4891c-ecef-404d-91a8-d730d1ecddb8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:35:58.986703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc4891c-ecef-404d-91a8-d730d1ecddb8 service neutron] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:35:58.998062 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1467/2922] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:35:58 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:36:07.760894 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=40 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:36:08.992532 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=12 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:36:09.977972 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a1cba7-eea6-431c-84bc-d84a8c89c784 service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:36:09.987075 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a1cba7-eea6-431c-84bc-d84a8c89c784 service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:36:09.990576 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5716d816-0def-400f-91c5-74fbf7ed75a5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cf29a8da103b411fbc79004eb371c05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:36:09.991159 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5716d816-0def-400f-91c5-74fbf7ed75a5 service neutron] REQUEST_METHOD: `GET` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:36:09.991433 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5716d816-0def-400f-91c5-74fbf7ed75a5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:36:09.991731 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5716d816-0def-400f-91c5-74fbf7ed75a5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:36:09.993414 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-5716d816-0def-400f-91c5-74fbf7ed75a5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:36:10.057166 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5716d816-0def-400f-91c5-74fbf7ed75a5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:36:10.058684 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5716d816-0def-400f-91c5-74fbf7ed75a5 service neutron] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:36:10.068091 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1456/2923] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:36:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:36:16.035846 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc4891c-ecef-404d-91a8-d730d1ecddb8 service neutron] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:36:16.038703 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-7bc4891c-ecef-404d-91a8-d730d1ecddb8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:36:16.055490 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc4891c-ecef-404d-91a8-d730d1ecddb8 service neutron] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:36:16.061074 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23bfefba-2ac5-43a2-95cf-2a6ed2d7a2e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:36:16.061721 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bfefba-2ac5-43a2-95cf-2a6ed2d7a2e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:36:16.062222 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bfefba-2ac5-43a2-95cf-2a6ed2d7a2e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:36:16.062661 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bfefba-2ac5-43a2-95cf-2a6ed2d7a2e2 admin admin] PATH_INFO: `/v3/users/44a710d801334b20b9f433eaad61cbc6` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:36:16.063764 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23bfefba-2ac5-43a2-95cf-2a6ed2d7a2e2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=44a710d801334b20b9f433eaad61cbc6)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:36:16.065714 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23bfefba-2ac5-43a2-95cf-2a6ed2d7a2e2 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:36:16.115509 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-23bfefba-2ac5-43a2-95cf-2a6ed2d7a2e2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '44a710d801334b20b9f433eaad61cbc6'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:36:16.123265 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-23bfefba-2ac5-43a2-95cf-2a6ed2d7a2e2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '44a710d801334b20b9f433eaad61cbc6'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:36:16.123957 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-23bfefba-2ac5-43a2-95cf-2a6ed2d7a2e2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '44a710d801334b20b9f433eaad61cbc6'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:36:16.131650 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-23bfefba-2ac5-43a2-95cf-2a6ed2d7a2e2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '44a710d801334b20b9f433eaad61cbc6'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:36:16.145904 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-23bfefba-2ac5-43a2-95cf-2a6ed2d7a2e2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '44a710d801334b20b9f433eaad61cbc6'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:36:16.152545 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1468/2924] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:36:16 2022] DELETE /identity/v3/users/44a710d801334b20b9f433eaad61cbc6 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:36:16.323084 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5716d816-0def-400f-91c5-74fbf7ed75a5 service neutron] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:36:16.333480 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5716d816-0def-400f-91c5-74fbf7ed75a5 service neutron] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:36:16.390793 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd3a726-74c4-4a6e-a8e7-61a5b698f33d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:36:16.391418 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd3a726-74c4-4a6e-a8e7-61a5b698f33d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:36:16.391699 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd3a726-74c4-4a6e-a8e7-61a5b698f33d admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:36:16.391971 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd3a726-74c4-4a6e-a8e7-61a5b698f33d admin admin] PATH_INFO: `/v3/projects/53d86b6e8a8343fca03ca6d38200f23f` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:36:16.393069 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd3a726-74c4-4a6e-a8e7-61a5b698f33d admin admin] RBAC: Authorizing `identity:delete_project(project_id=53d86b6e8a8343fca03ca6d38200f23f)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:36:16.394970 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd3a726-74c4-4a6e-a8e7-61a5b698f33d admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:36:16.442079 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-7cd3a726-74c4-4a6e-a8e7-61a5b698f33d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '53d86b6e8a8343fca03ca6d38200f23f', 'request_id': 'req-7cd3a726-74c4-4a6e-a8e7-61a5b698f33d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:36:16.448017 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-7cd3a726-74c4-4a6e-a8e7-61a5b698f33d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '53d86b6e8a8343fca03ca6d38200f23f', 'request_id': 'req-7cd3a726-74c4-4a6e-a8e7-61a5b698f33d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:36:16.453998 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-7cd3a726-74c4-4a6e-a8e7-61a5b698f33d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '53d86b6e8a8343fca03ca6d38200f23f', 'request_id': 'req-7cd3a726-74c4-4a6e-a8e7-61a5b698f33d'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:36:16.466741 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-7cd3a726-74c4-4a6e-a8e7-61a5b698f33d admin admin] The token cache is being invalidate because project 53d86b6e8a8343fca03ca6d38200f23f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:36:16.467028 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-7cd3a726-74c4-4a6e-a8e7-61a5b698f33d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:36:16.467647 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-7cd3a726-74c4-4a6e-a8e7-61a5b698f33d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:36:16.469112 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1457/2925] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:36:16 2022] DELETE /identity/v3/projects/53d86b6e8a8343fca03ca6d38200f23f => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:36:16.473266 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23bfefba-2ac5-43a2-95cf-2a6ed2d7a2e2 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:36:16.474694 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-23bfefba-2ac5-43a2-95cf-2a6ed2d7a2e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:36:16.488226 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23bfefba-2ac5-43a2-95cf-2a6ed2d7a2e2 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:36:16.544799 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-796f8f5a-ead5-419c-9ef2-f52d7e388030 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:36:16.545879 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796f8f5a-ead5-419c-9ef2-f52d7e388030 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:36:16.546387 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796f8f5a-ead5-419c-9ef2-f52d7e388030 admin admin] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:36:16.546882 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796f8f5a-ead5-419c-9ef2-f52d7e388030 admin admin] PATH_INFO: `/v3/users/e218879cb193406c9339b38dd833d929` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:36:16.548562 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-796f8f5a-ead5-419c-9ef2-f52d7e388030 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e218879cb193406c9339b38dd833d929)` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:36:16.551733 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-796f8f5a-ead5-419c-9ef2-f52d7e388030 admin admin] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:36:16.596243 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-796f8f5a-ead5-419c-9ef2-f52d7e388030 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e218879cb193406c9339b38dd833d929'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:36:16.604154 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-796f8f5a-ead5-419c-9ef2-f52d7e388030 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e218879cb193406c9339b38dd833d929'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:36:16.604744 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-796f8f5a-ead5-419c-9ef2-f52d7e388030 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e218879cb193406c9339b38dd833d929'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:36:16.612356 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-796f8f5a-ead5-419c-9ef2-f52d7e388030 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e218879cb193406c9339b38dd833d929'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:36:16.620903 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:36:16.626472 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.notifications [None req-796f8f5a-ead5-419c-9ef2-f52d7e388030 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e218879cb193406c9339b38dd833d929'} {{(pid=81420) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:36:16.633241 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1469/2926] 10.222.0.26 () {58 vars in 1224 bytes} [Tue Sep 27 04:36:16 2022] DELETE /identity/v3/users/e218879cb193406c9339b38dd833d929 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:36:16.800649 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd3a726-74c4-4a6e-a8e7-61a5b698f33d admin admin] Authenticating user token {{(pid=81421) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:36:16.802102 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.fernet_utils [None req-7cd3a726-74c4-4a6e-a8e7-61a5b698f33d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81421) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:36:16.816341 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd3a726-74c4-4a6e-a8e7-61a5b698f33d admin admin] Validating token access rules against request {{(pid=81421) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:36:16.837222 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:36:16.874659 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc37af0-bb57-4edb-a45a-51cecf2869ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6190ca8b8acd4e99b228f0851e2ed5af', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cc5c8e99cb64893a8d035d6aa4f3eb6', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81421) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:36:16.875218 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc37af0-bb57-4edb-a45a-51cecf2869ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:36:16.875495 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc37af0-bb57-4edb-a45a-51cecf2869ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:36:16.875765 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc37af0-bb57-4edb-a45a-51cecf2869ce admin admin] PATH_INFO: `/v3/projects/8069df7a2e984db0ac8925e1f0c242d8` {{(pid=81421) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:36:16.876858 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc37af0-bb57-4edb-a45a-51cecf2869ce admin admin] RBAC: Authorizing `identity:delete_project(project_id=8069df7a2e984db0ac8925e1f0c242d8)` {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:36:16.878765 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc37af0-bb57-4edb-a45a-51cecf2869ce admin admin] RBAC: Authorization granted {{(pid=81421) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:36:16.927905 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-bbc37af0-bb57-4edb-a45a-51cecf2869ce admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8069df7a2e984db0ac8925e1f0c242d8', 'request_id': 'req-bbc37af0-bb57-4edb-a45a-51cecf2869ce'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:36:16.933489 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-bbc37af0-bb57-4edb-a45a-51cecf2869ce admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8069df7a2e984db0ac8925e1f0c242d8', 'request_id': 'req-bbc37af0-bb57-4edb-a45a-51cecf2869ce'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:36:16.939553 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-bbc37af0-bb57-4edb-a45a-51cecf2869ce admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8069df7a2e984db0ac8925e1f0c242d8', 'request_id': 'req-bbc37af0-bb57-4edb-a45a-51cecf2869ce'} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:36:16.951269 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-bbc37af0-bb57-4edb-a45a-51cecf2869ce admin admin] The token cache is being invalidate because project 8069df7a2e984db0ac8925e1f0c242d8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81421) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 04:36:16.951558 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-bbc37af0-bb57-4edb-a45a-51cecf2869ce admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:36:16.952180 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG keystone.notifications [None req-bbc37af0-bb57-4edb-a45a-51cecf2869ce admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81421) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 04:36:16.953648 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: [pid: 81421|app: 0|req: 1458/2927] 10.222.0.26 () {58 vars in 1233 bytes} [Tue Sep 27 04:36:16 2022] DELETE /identity/v3/projects/8069df7a2e984db0ac8925e1f0c242d8 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 04:36:26.631529 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=3,keystone:DELETE=1 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:36:26.948400 dv-cinder-859186-3-iscsi devstack@keystone.service[81421]: DEBUG dbcounter [-] [81421] Writing DB stats keystone:SELECT=40,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81421) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 04:38:16.855889 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-796f8f5a-ead5-419c-9ef2-f52d7e388030 admin admin] Authenticating user token {{(pid=81420) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 04:38:16.857836 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-796f8f5a-ead5-419c-9ef2-f52d7e388030 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:38:16.999026 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-796f8f5a-ead5-419c-9ef2-f52d7e388030 admin admin] Validating token access rules against request {{(pid=81420) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 04:38:17.058082 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75cdc41b-b53f-4c9f-bcf1-1e5ce7167757 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3a1bc198dfc0486e9e56d4fb28cda05d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c830a3bfece6456486f4b094076b5364', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81420) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 04:38:17.058686 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cdc41b-b53f-4c9f-bcf1-1e5ce7167757 service placement] REQUEST_METHOD: `GET` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 04:38:17.058961 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cdc41b-b53f-4c9f-bcf1-1e5ce7167757 service placement] SCRIPT_NAME: `/identity` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 04:38:17.059231 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cdc41b-b53f-4c9f-bcf1-1e5ce7167757 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81420) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 04:38:17.060997 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.fernet_utils [None req-75cdc41b-b53f-4c9f-bcf1-1e5ce7167757 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81420) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 04:38:17.071911 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75cdc41b-b53f-4c9f-bcf1-1e5ce7167757 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 04:38:17.073470 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75cdc41b-b53f-4c9f-bcf1-1e5ce7167757 service placement] RBAC: Authorization granted {{(pid=81420) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 04:38:17.083695 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: [pid: 81420|app: 0|req: 1470/2928] 10.222.0.26 () {60 vars in 1365 bytes} [Tue Sep 27 04:38:16 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 228 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 04:38:27.079420 dv-cinder-859186-3-iscsi devstack@keystone.service[81420]: DEBUG dbcounter [-] [81420] Writing DB stats keystone:SELECT=89 {{(pid=81420) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}